Automatically canceled job continues execution and fails

Summary

When a job gets canceled mid run due to some other job failing and workflow.auto_cancel.on_job_failure being set to all, the script isn't always canceled cleanly. The currently running command in script fails and after_script is executed as if the job failed, even though UI correctly says that the job was canceled.

Steps to reproduce

In our case a lint and sempgrep jobs are running concurrently. If lint fails it cancels semgrep job while it is running but the script continues execution, trying to grep the output file that never gets generated.

The minimal config here isn't 1:1 with the problem because I couldn't manage to reproduce it but it still demonstrates how the job doesn't cancel properly but continues execution.

.gitlab-ci.yml

workflow:
  auto_cancel:
    on_job_failure: all

stages:
  - lint

lint:
  stage: lint
  image: registry.gitlab.com/pipeline-components/ruff:0.18.0
  script: 
    - sleep 5
    - exit 1
  interruptible: true

semgrep-mr:
  stage: lint
  image: returntocorp/semgrep
  script:
    # this is just an attempt at simulating the problem. the actual script is something akin to
    # - |
    #   if ! semgrep ci --verbose --no-suppress-errors --output semgrep.xml --junit-xml; then
    #     grep "failure type" semgrep.xml
    #     exit 1
    #   fi
    - set -euo pipefail
    - |
      if ! sleep 50; then
        echo "this should not print"
        exit 1
      fi
    - echo "this should only print if the job succeeds, which it shouldn't"
  after_script:
    - |
      echo "\$CI_JOB_STATUS == $CI_JOB_STATUS"
      [ "${CI_JOB_STATUS:-canceled}" == "canceled" ] && exit 0
      echo "this should not print either"
  interruptible: true

Actual behavior

semgrep-mr continues execution even though UI says it's canceling. "this should only print if the job succeeds, which it shouldn't" and "this should not print either" show up in the job log. Job log doesn't show anywhere that the job was canceled.

Expected behavior

Job log should only say that it got canceled and nothing should print there.

Relevant logs and/or screenshots

job log

Running with gitlab-runner 17.0.0 (44feccdf)
  on Shared runner g-JwDap9, system ID: r_g3Tb5m4hnDY2
Resolving secrets
section_start:1721301512:prepare_executor
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-shared-runner
Using Kubernetes executor with image returntocorp/semgrep ...
Using attach strategy to execute scripts...
section_end:1721301512:prepare_executor
section_start:1721301512:prepare_script
Preparing environment
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...
Waiting for pod gitlab-shared-runner/runner-g-jwdap9-project-1-concurrent-0-lj5t08e1 to be running, status is Pending
Running on runner-g-jwdap9-project-1-concurrent-0-lj5t08e1 via gitlab-shared-runner-gitlab-runner-557c9cbbb-qlggg...

section_end:1721301516:prepare_script
section_start:1721301516:get_sources
Getting source from Git repository
Fetching changes with git depth set to 10...
Initialized empty Git repository in /builds/custobar/custobar/.git/
Created fresh repository.
Checking out d451d77d as detached HEAD (ref is repro-canceled-job-after-script-bug)...

Skipping Git submodules setup

section_end:1721301522:get_sources
section_start:1721301522:step_script
Executing "step_script" stage of the job script
$ set -euo pipefail
$ if ! sleep 50; then # collapsed multi-line command
$ echo "this should only print if the job succeeds, which it shouldn't"
this should only print if the job succeeds, which it shouldn't

section_end:1721301573:step_script
section_start:1721301573:after_script
Running after_script
Running after script...
$ echo "\$CI_JOB_STATUS == $CI_JOB_STATUS" # collapsed multi-line command
$CI_JOB_STATUS == success
this should not print either

section_end:1721301573:after_script
section_start:1721301573:cleanup_file_variables
Cleaning up project directory and file based variables

section_end:1721301574:cleanup_file_variables
Job succeeded

Environment description

Kubernetes executor on a self hosted Gitlab instance.

config.toml contents

config.template.toml
[[runners]]
  [runners.cache]
    Type = "gcs"
    Path = ""
    Shared = true
    [runners.cache.gcs]
      BucketName = "not-relevant-here"

config.toml
shutdown_timeout = 0
concurrent = 4
check_interval = 1
log_level = "info"

Used GitLab Runner version

Running with gitlab-runner 17.0.0 (44feccdf)
  on Shared runner g-JwDap9, system ID: r_g3Tb5m4hnDY2
Resolving secrets
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-shared-runner
Using Kubernetes executor with image returntocorp/semgrep ...

Possible fixes