Use fresh context in StopKillWait when build ctx is cancelled

Summary

When a job is cancelled, Abort() fires and cancels the build-level context. If that cancellation races with the cleanup path, StopKillWait receives an already-cancelled ctx. The retryWait loop condition ctx.Err() == nil exits immediately — ContainerStop is never called, and PID 1 never receives SIGTERM. It only gets SIGKILL after the network is already disconnected.

This was identified in https://gitlab.com/gitlab-com/request-for-help/-/work_items/4455 where DAP workflows observed WebSocket close code 1006 (abnormal closure) on job cancellation, caused by the network being torn down before the container process had a chance to shut down gracefully.

Root cause

StopKillWait in executors/docker/internal/wait/wait.go passes the build context ctx to both retryWait and ContainerStop. The graceful-exit func (the SIGTERM kill script) already correctly used context.Background() — but ContainerStop did not.

Observed docker events sequence without fix:

exec_create    ← SIGTERM kill script (own ctx, always runs)
exec_die=137   ← kill script force-killed
               ← retryWait bails: build ctx cancelled, ContainerStop skipped
network disc.  ← removeContainer → disconnectNetwork
container die  ← ContainerRemove(Force) → SIGKILL

PID 1 never gets SIGTERM, only SIGKILL after the network is already gone — which is why the WebSocket closes with code 1006 instead of 1000.

Testing

  • Added TestDockerWaiter_StopKillWait_CancelledContext which pre-cancels the context before calling StopKillWait and asserts that ContainerStop is still called.
  • Manual verification: build locally with make runner-and-helper-bin-host, point GDK at the binary, set feature flag FF_SKIP_DOCKER_KILL_SCRIPT to true, cancel a running duo job, observe docker events to confirm container stop appears before network disconnect.
Edited by Halil Coban

Merge request reports

Loading