Report exit status for missing predefined-stage scripts
Closes https://gitlab.com/gitlab-org/gitlab-runner/-/issues/39606
!6941 (merged) fixed the attach-strategy hang for build-container stages (step_script, detect_shell_script) by capturing the detect shell's exit code from outside the generated script and emitting a fallback JSON marker when the script file is missing. It never covered predefined stages (get_sources, restore_cache, cleanup_file_variables, ...), which run through the helper container's gitlab-runner-build entrypoint instead - this MR closes that remaining gap.
gitlab-runner-build (baked into the helper image, see dockerfiles/runner-helper/scripts/gitlab-runner-build in gitlab-org/ci-cd/runner-tools/base-images) is just:
#!/bin/bash
umask 0000
exec /bin/bashSince it's invoked as a here-string (gitlab-runner-build <<< /stage/script/path.sh), the exec'd bash receives the literal path string as stdin and tries to run it as a one-line command. If the script is missing, bash exits non-zero but never prints the JSON completion marker the trap inside the script would normally emit, so remoteProcessTerminated waits forever - the same hang as the build-container case, just via an opaque entrypoint.
- Add
fallbackExitStatusMarkerPredefinedFmt, mirroring the existing build-container fallback marker but for the here-string formgitlab-runner-buildneeds - Wrap the predefined-stage branch in
getContainerInfowith the same externally-captured exit code + fallback marker technique - Extend the existing
TestGetContainerInfoandTestGetContainerInfoReportsExitStatusWhenStageScriptMissingtests with predefined-stage cases, including one that runs the real generated command through a real shell to prove a missing script now reports exit 127 instead of hanging
Doesn't touch the exec legacy execution strategy (FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=true), which already gets real exit codes via the Kubernetes pods/exec subresource and was never affected by this bug.
Local verification against a real cluster
Reproduced the actual race from https://gitlab.com/gitlab-org/gitlab-runner/-/issues/39606 end to end against a live local Kubernetes cluster, rather than only the in process bash simulation the unit tests use. A background watcher planted in the build container's persistent shell continuously deletes cleanup_file_variables's script the instant it appears on the shared scripts volume, racing the runner's own save then exec sequence for that stage.
- On
origin/mainat https://gitlab.com/gitlab-org/gitlab-runner/-/blob/e306db172910e79a7741b0695dd9c2bb7ce9eee7/executors/kubernetes/kubernetes.go#L3265 (pre-fix): ran the identical test 5 times. On the run where the watcher won the race,build.Run()hung completely, killed only by the test binary's own timeout. The goroutine dump pinpoints the hang exactly at that line, the bare<-s.remoteProcessTerminatedchannel receive. The job's entire trace buffer was lost since nothing ever unblocked to flush it, matching real world reports of jobs going silently stuck. - On this MR's branch (fixed): 4 out of 4 clean runs, plus one run where the race demonstrably fired (
/bin/bash: line 1: /scripts-0-0/cleanup_file_variables: No such file or directoryappeared in the trace, proving the deletion won) and the job still completed successfully in about 7 seconds via the fallback marker.