Fix proxy-exec and step-runner hangs when scripts spawn background processes

Arran Walkerrequested to merge
ajwalker/fix-exec into main

What

Fixes a class of hang where commands spawned by proxy-exec, step-runner's script execution, and script_legacy's executor would block indefinitely if the script started a background process ((daemon &), nohup foo &, etc.).

Root cause

When cmd.Stdout / cmd.Stderr are non-*os.File writers, os/exec creates internal pipes and copy goroutines. cmd.Run() / cmd.Wait() waits for those goroutines to see EOF. A backgrounded grandchild inherits the pipe write-end on fork, keeps it open after the direct child exits, EOF never arrives, and the job hangs until the grandchild eventually exits (or forever).

Changes

  • commands/helpers/proxy_exec.go — set cmd.WaitDelay. proxy-exec stays semantically transparent (matches sh -c behaviour for orphaned processes); WaitDelay just bounds the pipe drain after the direct child exits.
  • commands/steps/steps.go and functions/script_legacy/internal/executor.go — switch to gracefulexitcmd.New(...). Step boundaries are a meaningful cleanup point, so we now run the script in its own process group, send SIGTERM on context cancel, and SIGKILL the group after Wait() to reap any survivors. On Windows, gracefulexitcmd uses Job Objects for the equivalent guarantee.

Behaviour notes

  • proxy-exec: orphaned processes survive (as they would under sh -c); their stdout/stderr stops being captured the moment the direct child exits.
  • step-runner / script_legacy: orphaned processes are now killed when the step exits. CI scripts relying on cross-step process survival via (cmd &) would need to use a different mechanism (services, sidecar containers, etc.).

Tests

  • New TestProxyExecBackgroundProcess regression test reproduces the hang against unmodified code (fails at 5s deadline) and passes in ~1s with the fix. Skipped on Windows (uses POSIX double-fork semantics).
  • Existing TestServe/with_context_being_canceled_from_the_outside updated from signal: killed to signal: terminated to reflect the new graceful SIGTERM-first behaviour.
  • All other existing tests in the affected packages pass unchanged.

Merge request reports