Report exit status for missing predefined-stage scripts

Closes https://gitlab.com/gitlab-org/gitlab-runner/-/issues/39606

!6941 (merged) fixed the attach-strategy hang for build-container stages (step_script, detect_shell_script) by capturing the detect shell's exit code from outside the generated script and emitting a fallback JSON marker when the script file is missing. It never covered predefined stages (get_sources, restore_cache, cleanup_file_variables, ...), which run through the helper container's gitlab-runner-build entrypoint instead - this MR closes that remaining gap.

gitlab-runner-build (baked into the helper image, see dockerfiles/runner-helper/scripts/gitlab-runner-build in gitlab-org/ci-cd/runner-tools/base-images) is just:

#!/bin/bash
umask 0000
exec /bin/bash

Since it's invoked as a here-string (gitlab-runner-build <<< /stage/script/path.sh), the exec'd bash receives the literal path string as stdin and tries to run it as a one-line command. If the script is missing, bash exits non-zero but never prints the JSON completion marker the trap inside the script would normally emit, so remoteProcessTerminated waits forever - the same hang as the build-container case, just via an opaque entrypoint.

  • Add fallbackExitStatusMarkerPredefinedFmt, mirroring the existing build-container fallback marker but for the here-string form gitlab-runner-build needs
  • Wrap the predefined-stage branch in getContainerInfo with the same externally-captured exit code + fallback marker technique
  • Extend the existing TestGetContainerInfo and TestGetContainerInfoReportsExitStatusWhenStageScriptMissing tests with predefined-stage cases, including one that runs the real generated command through a real shell to prove a missing script now reports exit 127 instead of hanging

Doesn't touch the exec legacy execution strategy (FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=true), which already gets real exit codes via the Kubernetes pods/exec subresource and was never affected by this bug.

Local verification against a real cluster

Reproduced the actual race from https://gitlab.com/gitlab-org/gitlab-runner/-/issues/39606 end to end against a live local Kubernetes cluster, rather than only the in process bash simulation the unit tests use. A background watcher planted in the build container's persistent shell continuously deletes cleanup_file_variables's script the instant it appears on the shared scripts volume, racing the runner's own save then exec sequence for that stage.

  • On origin/main at https://gitlab.com/gitlab-org/gitlab-runner/-/blob/e306db172910e79a7741b0695dd9c2bb7ce9eee7/executors/kubernetes/kubernetes.go#L3265 (pre-fix): ran the identical test 5 times. On the run where the watcher won the race, build.Run() hung completely, killed only by the test binary's own timeout. The goroutine dump pinpoints the hang exactly at that line, the bare <-s.remoteProcessTerminated channel receive. The job's entire trace buffer was lost since nothing ever unblocked to flush it, matching real world reports of jobs going silently stuck.
  • On this MR's branch (fixed): 4 out of 4 clean runs, plus one run where the race demonstrably fired (/bin/bash: line 1: /scripts-0-0/cleanup_file_variables: No such file or directory appeared in the trace, proving the deletion won) and the job still completed successfully in about 7 seconds via the fallback marker.
Edited by Lachlan Grant

Merge request reports

Loading
Loading