Increase Docker stop timeout to 2m and improve error messaging

Daniel Keenanrequested to merge
adjust-container-stop-handling into main

What does this MR do?

  • Increases the Docker container stop-kill timeout from 30s to 2m.
  • Improves error messaging surfaced when this timeout is reached.

Why was this MR needed?

There have been several reports of intermittent job failures since updating to Runner v19.1.0.

Before v19.1.0, there was effectively no timeout on the container stop-kill operation (which is executed at the end of each job stage to prepare the container for the next one).

In v19.1.0, this change introduced a 30s timeout.

The 30s timeout seems reasonable, but the reported job failures appear to match the failure mode resulting from containers not being stopped/killed within 30s. Slow shutdown may be caused by pressure on the Docker daemon on a busy host or one with a lot of accumulated state.

While the above is only an informed hypothesis, this MR achieves two things:

  1. Increases the timeout to 2m (a 4× increase), giving a generous amount of additional headroom for containers to be stopped/killed. This duration is a judgement call, made in the absence of more concrete data.
  2. Improves the error message returned:
    • Before: ERROR: Job failed (system failure): context deadline exceeded
    • After: ERROR: Job failed (system failure): container stop-kill wait timed out after 2m0s

This will help us verify or rule out this code path as the cause of the intermittent failures reported.

What's the best way to test this MR?

  • Simulate a container not stopping within 2m by temporarily adding a line such as time.Sleep(3 * time.Minute) to dockerWaiter.StopKillWait
  • Run a simple test job using the Docker executor and check for ERROR: Job failed (system failure): container stop-kill wait timed out after 2m0s as the last line of the output.

What are the relevant issue numbers?

Edited by Daniel Keenan

Merge request reports