## Introduction This issue was created by Duo while investigating possible causes for a specific Maven job failing to run `after_script` tasks when the job is cancelled ([ZD internal link](https://gitlab.zendesk.com/agent/tickets/714354)). Testing showed the problem occurs when the job is cancelled while processing certain intensive tasks in the job script, with the job just aborting straight into the `canceled` state with nothing further captured in the job log. Please validate Duo's findings and advise whether this is a known issue and whether any workarounds exist. ## Summary When a job is cancelled, there is a race between the runner's `processLogs` goroutine (which exits as soon as the job's context is cancelled) and the actual termination of the running build process plus the write of its exit status JSON (`{"command_exit_code": ..., "script": ...}`) to the log file. If the build process takes too long to terminate after receiving `SIGTERM`, `processLogs` can exit before the exit status is written, causing `after_script` to be skipped. This has been observed intermittently with a customer running Maven builds on the Kubernetes executor, but the underlying race is generic and could reproduce with any workload that terminates slowly. ## Root cause factors ### 1. Cancellation script only kills direct children `stageCancellationScript` in `kubernetes.go` generates a script that walks `/proc` and kills only the **direct children** of the stage script process: ``` kill -TERM $(for item in $(...<scriptName>...) do \ test -f /proc/${item}/task/${item}/children && \ cat /proc/${item}/task/${item}/children && echo; done) 2> /dev/null ``` For workloads with deep process trees (e.g. shell → Maven wrapper → Maven JVM → plugin JVMs → forked compiler/test processes), the JVM is often not a direct child of the stage script. `SIGTERM` may not reach it at all, or only reaches it late after propagating down through wrapper scripts. ### 2. Slow JVM shutdown under I/O load When Maven is actively resolving SNAPSHOT dependencies (many concurrent HTTP connections), the JVM's shutdown hooks take noticeably longer to run once `SIGTERM` is eventually delivered, since many threads are blocked on network I/O. This further delays process exit and the subsequent write of the exit status JSON by the `runner_script_trap` bash trap. ### 3. `processLogs` has zero tolerance for the delay `processLogs`'s context is cancelled immediately when the cancel signal arrives, and it exits on `ctx.Done()` with no grace period to wait for a pending exit-status write that may already be in flight. ## Why it's intermittent The outcome depends on a race between: - How quickly the process tree actually terminates and the exit status JSON is written to the log file, vs. - How quickly the `processLogs` goroutine's `ctx.Done()` case fires. For fast-terminating workloads (simple shell scripts, Go binaries) this window is milliseconds and essentially unobservable. For workloads with deep process trees and slow, I/O-bound shutdown (JVM-based tools like Maven/Gradle being a prominent example), the window widens to potentially several seconds, making the race reliably observable. ## Suggested areas to investigate - Whether the cancellation script should recursively walk the full descendant process tree instead of only direct children (`/proc/*/task/*/children` is single-level). - Whether `FF_USE_DUMB_INIT_WITH_KUBERNETES_EXECUTOR` improves signal propagation enough to reduce (not eliminate) the frequency of this race, and whether it should be considered more broadly. - Whether `processLogs` (or the log-processing path generally) should allow a bounded grace period after cancellation to observe a pending exit-status write already started before flagging the stage as terminated without status. - Whether the fundamental issue is the unsynchronised lifetime of `processLogs`'s context relative to the actual write of the exit status file, independent of executor or workload — i.e. this is not Kubernetes-executor-specific in principle, though the process-tree issue is most acute there. ## Impact `after_script` is expected to run even on cancelled/failed jobs (e.g. for cleanup, notifications, artifact upload). Silently skipping it on cancellation for slow-shutdown workloads can leave environments in an inconsistent state without any indication to the user. ## Reproduction notes Reported by a customer running Maven builds on the Kubernetes executor. Cancellations occurred while Maven was actively downloading SNAPSHOT dependencies. Reproduction is timing-dependent; a deterministic reproduction would need a test process that: - Has a process tree at least 2 levels deep relative to the stage script, and - Delays exit (and exit-status write) after receiving `SIGTERM` by an amount comparable to or greater than the time it takes `processLogs` to observe cancellation.