Gracefully halt cancelled jobs
Problem to solve
When a job is cancelled in the GitLab UI, the runner sends a SIGKILL to the job, preventing it from cleaning up resources and causing resource leaks.
Steps to reproduce
Cancel a job that uses external resources.
Actual behavior
The SIGKILL halts the job gracelessly.
Expected behavior
Using a SIGTERM would halt the job and allow for gracefully shutting down with resource cleanup. If the SIGTERM is not responded to in a reasonable time (10s) then a subsequent SIGKILL is sent to terminate the process.
Relevant logs and/or screenshots
SIGTERM is the termination signal. The default behavior is to terminate the process, but it also can be caught or ignored. The intention is to kill the process, gracefully or not, but to first allow it a chance to cleanup.
SIGKILL is the kill signal. The only behavior is to kill the process, immediately. As the process cannot catch the signal, it cannot cleanup, and thus this is a signal of last resort.
Environment description
We are using a shell environment for our runner jobs.
Used GitLab Runner version
9.2.0