Kill pipelines that hang after the request timeout

Miroslav Vadkertirequested to merge
kill-hung-pipelines into main

The job scripts run citool under timeout --signal=SIGUSR1 without --kill-after. When citool or podman run does not exit after the signal, timeout waits forever and the Nomad job never finishes.

On the public ranch, tf-tmt/dispatch-1791268670-c7e85b50 ran for 23 days this way. Its request 5b02ad93-77ec-4674-90e7-7c5b7a6189c0 timed out after 720 minutes on 2026-10-06 and citool reported it as canceled, but the process stayed alive and kept 3 GB reserved on testing-farm-worker-69396fe2. A Nomad job stop did not kill it either.

Changes in all 8 job scripts (worker-{public,redhat}/jobs/tf-{sti,tmt,tmt-container,tmt-multihost}):

  • --kill-after=1h makes timeout send SIGKILL one hour after the SIGUSR1. citool needed 34 minutes to clean up in the case above.
  • With --foreground, timeout kills only podman run, and the container keeps running under conmon. podman rm --force after timeout returns removes it. On a clean exit --rm has already removed the container and the command does nothing.

A hard-killed pipeline skips citool's own cleanup, so its Artemis guests can leak. Today the same pipeline hangs forever and holds the worker slot.

Generated-by: Claude Code

Edited by Miroslav Vadkerti

Merge request reports