Fail fast on a missing build-container stage script (attach strategy)

Problem

Kubernetes executor jobs using the attach strategy intermittently hang for hours: a stage script (e.g. step_script) goes missing from the emptyDir volume, the shell reports No such file or directory, and the job then just sits there sending identical job-status=running heartbeats until it hits the job timeout / activeDeadlineSeconds, instead of failing promptly.

Reported in https://gitlab.com/gitlab-org/gitlab-runner/-/issues/39354 and https://gitlab.com/gitlab-com/request-for-help/-/work_items/4934.

Root cause

The attach strategy detects a stage's completion purely by scanning the job log for a JSON marker ({"command_exit_code": N, ...}) printed by a trap ... EXIT installed inside the generated stage script itself (bashJSONTerminationScript in shells/bash.go). The detect-shell wrapper does exec bash <stage-script-path>; if that file can't be opened, bash exits 127 immediately, before it ever reads a line of the script - so the trap never installs and no marker is ever produced.

runInContainer then blocks forever waiting for that marker via remoteProcessTerminated. Nothing else catches this: the pod/container stay healthy (it's just one backgrounded command inside them that exited), and the log file still exists (so the "log file deleted" safeguard from !2824 (merged) doesn't fire either). The only thing that eventually ends the job is the plain job timeout.

There is no save/attach race - saveScriptOnEmptyDir is fully synchronous and completes before the attach call runs.

Fix

Capture the exit code of the detect-shell invocation from outside the script and print a fallback JSON marker whenever it comes back non-zero. The trap always exits 0 once it does get to run, so the two paths can't collide or double-report a status for the same invocation.

What this does not fix

This only covers stages that run in the build container (e.g. step_script, detect_shell_script). Predefined stages that run in the helper container via the gitlab-runner-build entrypoint (get_sources, restore_cache, cleanup_file_variables - the exact stages named in both linked issues) go through a different mechanism. That entrypoint is a small script baked into the helper base image, and its canonical source isn't part of this Go module, so the equivalent guard needs to be applied wherever it's actually maintained.

Also unresolved: why the script goes missing from the emptyDir in the first place. That's an environmental/trigger question - the reporting customer's scale means live diagnostics (kubeSOS, ls -la /scripts-* during the hang) aren't available to confirm a cause. This MR removes the customer-visible impact (multi-hour silent hang) regardless of that trigger, turning it into an immediate, clear job failure - but doesn't stop the underlying file-missing event from happening again.

Testing

  • New unit test (TestGetContainerInfoReportsExitStatusWhenStageScriptMissing) runs the actual generated command through a real shell against both a missing and a present script, and asserts the fallback marker appears only in the missing case (no double-marker regression for the normal path).
  • Existing executors/kubernetes test suite passes (two pre-existing, architecture-specific failures unrelated to this change, confirmed present on unmodified main as well).
Edited by Lachlan Grant

Merge request reports

Loading