Skip to content

Jobs appear to hang when cancelled in Kubernetes runner

Summary

When using a Kubernetes executor with a self-hosted Runner (16.11.1) on GitLab.com jobs appear to hang instead of cancelling immediately.

Steps to reproduce

Bash

gitlab-ci
variables:
  FF_RETRIEVE_POD_WARNING_EVENTS: "true"
  FF_PRINT_POD_EVENTS: "true"
  FF_SCRIPT_SECTIONS: "true"
  FF_USE_DUMB_INIT_WITH_KUBERNETES_EXECUTOR: "true"

simple-job:
  script:
    - sleep 3600
config.toml
concurrent = 1
check_interval = 1
log_level = "debug"
shutdown_timeout = 0

listen_address = ':9252'

[session_server]
  session_timeout = 1800

[[runners]]
  name = "investigation"
  url = "https://gitlab.com/"
  id = 0
  token = "glrt-REDACTED"
  token_obtained_at = "0001-01-01T00:00:00Z"
  token_expires_at = "0001-01-01T00:00:00Z"
  executor = "kubernetes"
  shell = "bash"
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = "alpine"
    pod_termination_grace_period_seconds = 3600
    namespace = ""
    namespace_overwrite_allowed = ""
    pod_labels_overwrite_allowed = ""
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    node_selector_overwrite_allowed = ".*"
    allow_privilege_escalation = false
    [[runners.kubernetes.services]]
    [runners.kubernetes.dns_config]
    [runners.kubernetes.pod_labels]

Actual behavior

The job won't stop and will hang indefinitely

Expected behavior

The job should stop

Possible fixes