Skip to content

Kubernetes Executor outputs entirety of Pod environment and `step_script` when a job is cancelled

Summary

I recently noticed an issue when cancelling Kubernetes executor CI jobs (I think it started with Gitlab Runner 17.3.0): If I cancel a CI job before it completes, a termination dump is output in the after_script section of the job log which contains the entire contents of step_script. This is troubling because it contains the full environment with all variables expanded. This does not occur with shell executor Runners.

I believe this was introduced as part of a fix for k8s executor jobs failing to cancel. I noticed a note on this MR in which Romuald Atchadé mentions seeing the "Terminated" message in the job log, but I do not see any mention of a resolution. I think this was probably seen as a minor thing when testing with a container running BASH 5.x because BASH 5.x containers simply output the string "Terminated". However, containers running BASH 4.x dump the entirety of the environment and killed command after the "Terminated" message.

Steps to reproduce

  1. Run a CI job in a RHEL 8 container image (or any image using BASH 4.x) with a Kubernetes executor Runner on Gitlab Runner 17.3.1 with a long pause.
  2. Cancel the job before completion.
  3. Observe the bottom of the job log (after "Running after_script")
.gitlab-ci.yml 
testing_k8s:
  stage: test
  tags:
    - k8s
  image:
    name: registry.access.redhat.com/ubi8:latest
  script:
    - export TEST='foo'
    - sleep 400

Actual behavior

The BASH "Terminated" message is logged in the job log when it should be suppressed. This is especially troubling with BASH 4.x which also outputs the entire CI job script including all expanded environment variables.

Expected behavior

The termination should be silent.

Relevant logs and/or screenshots

job log 
Normal   Created   Created container helper
Normal   Started   Started container helper
Running on runner-yuvec-6a-project-1201-concurrent-0-23ms38so via gitlab-runners-gitlab-runner-9f554cf9c-p4v4m...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/ci-cd/infrastructure-tests/gitlab-runner-tests/.git/
Created fresh repository.
Checking out cd7d68df as detached HEAD (ref is master)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:31
$ export TEST='foo'
$ sleep 400
WARNING: script canceled externally (UI, API)
Running after_script
00:01
/scripts-1201-2753391/step_script: line 17:    29 Done                    :
        30 Terminated              | eval 'export FF_TEST_FEATURE=false
export FF_NETWORK_PER_BUILD=false
export FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false
export FF_USE_DIRECT_DOWNLOAD=true
export FF_SKIP_NOOP_BUILD_STAGES=true
export FF_USE_FASTZIP=false
export FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR=false
export FF_ENABLE_BASH_EXIT_CODE_CHECK=false
export FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY=false
export FF_USE_NEW_BASH_EVAL_STRATEGY=false
export FF_USE_POWERSHELL_PATH_RESOLVER=false
export FF_USE_DYNAMIC_TRACE_FORCE_SEND_INTERVAL=false
export FF_SCRIPT_SECTIONS=false
export FF_ENABLE_JOB_CLEANUP=false
export FF_KUBERNETES_HONOR_ENTRYPOINT=false
export FF_POSIXLY_CORRECT_ESCAPES=false
export FF_RESOLVE_FULL_TLS_CHAIN=false
export FF_DISABLE_POWERSHELL_STDIN=false
export FF_USE_POD_ACTIVE_DEADLINE_SECONDS=true
export FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=false
export FF_SET_PERMISSIONS_BEFORE_CLEANUP=true
export FF_SECRET_RESOLVING_FAILS_IF_MISSING=true
export FF_RETRIEVE_POD_WARNING_EVENTS=true
export FF_PRINT_POD_EVENTS=false
(TRUNCATED)

Environment description

Gitlab Onmibus EE Ultimate with Kubernetes 17.3.1 Gitlab Runners (Helm) on Kubernetes 1.29.x.

config.toml contents 
[[runners]]
  tls-ca-file = "/home/gitlab-runner/.gitlab-runner/certs/local.pem"
  output_limit = 30720
  [runners.kubernetes]
    helper_image = "gitlab01dockerreg.domain.com/container-images/gitlab-runner-helper/gitlab-runner-helper:latest"
    namespace = "{{.Release.Namespace}}"
    image_pull_secrets = ["local-regcred"]
    image = "alpine"

Used GitLab Runner version

Running with gitlab-runner 17.3.1 (66269445) Using Kubernetes executor with image registry.access.redhat.com/ubi8:latest

Possible fixes