Remove Docker stop-kill timeout and add duration logging

What does this MR do?

In the Docker executor, a Docker stop-kill/wait teardown operation is carried out at the end of each job stage, to prepare the relevant container (job container or helper container, depending on the stage) for the next stage.

This MR updates the operation's behaviour such that:

  1. There is no longer a timeout on the container stop/wait itself, so a slow-to-stop container no longer fails an otherwise-healthy job.
  2. The best-effort graceful SIGTERM step is kept time-bounded (so a stuck exec can't block teardown), while the stop itself runs unbounded to completion.
  3. The duration of the operation is logged (at debug level).

Why was this MR needed?

Following a regression in Runner v19.1.0, some users saw intermittently failing jobs as a result of a 30s timeout which had been introduced on the Docker stop-kill/wait operation (as part of a separate bug fix) where previously there was no timeout.

A mitigation was rolled out in a patch release, which confirmed the root cause. Some users reported that it fixed the issue, some reported improvement but still occasional failures, and for some the increased timeout remained insufficient.

The idea of making this timeout configurable was considered, but after discussion with @adebayo_a, we agreed that it would increase Runner's configuration surface for little or no benefit. More specifically, the configuration would likely confuse users and the timeout remaining in place by default would frustrate users who naturally don't want their jobs to fail altogether due to a teardown operation being slow (we've seen that typical durations can vary a lot across workloads and environments).

So, our answer to the problem is to remove the timeout altogether, restoring the no-teardown-timeout behaviour of the pre-v19.1.0 non-cancelled flow, while also adding debug logging of the operation duration. This improves observability so that operators can identify performance degradation, without blocking jobs completely through an arbitrary timeout.

Note: Detaching from the cancelled context (rather than skipping teardown) preserves the cleanup-on-cancellation behaviour that !6636 (merged) introduced: jobs cancelled while a container is still running still get SIGTERM and a chance to terminate gracefully. The graceful SIGTERM step is time-bounded so a stuck exec can't block teardown; the container stop itself is left unbounded and, should a daemon stall, is ultimately reaped by executor.Cleanup.

What's the best way to test this MR?

  1. Run gitlab-runner with the Docker executor and --log-level=debug and --log-format=json.
  2. Run a simple job.
  3. Verify the presence of lines like:
{"container_id":"e2e666b2f62e9b0d986fb06b2a957671c8aeb618f8c6bdddfed2e5f1770b714c","container_name":"runner-nmlhlg9iz-project-80368065-concurrent-0-e18629fbd7424a26-predefined","duration_s":0.054411917,"gitlab_user_id":35011548,"job":15273149422,"job_canceled":false,"level":"debug","msg":"Container stop-kill wait completed","name":"dkeenan--20260225-6J579","namespace_id":127367842,"organization_id":1,"pipeline_id":2623887386,"project":80368065,"project_full_path":"daniel-keenan-test-group/test","root_namespace_id":127367842,"runner":"nmlhlG9IZ","runner_name":"Test Runner Docker","time":"2026-07-09T20:11:13.696011+01:00"}

at the end of each relevant stage in the runner logs.

What are the relevant issue numbers?

Relates to #39554, #39556 (closed), #39555 (closed)

Edited by Daniel Keenan

Merge request reports

Loading