## Note (revised 2022-04-28) If you are still experiencing similar issues as described in this issue, then add a comment with your issue details to the [CI process does not receive SIGTERM on termination](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27443) issue. ## Overview I have a long running compile job (~40 minutes). I made 2 pushes one after another. I stopped the first running job with the ui. It tells me that the job is canceled. But the second job stays at pending. I suspect that the runner finishes the job and is not properly terminated. ~~Is there a way to test my hypothesis?~~ I'm using the shell executor for the runner (gitlab and runner are on ubuntu 16.04) Edit: as written in a comment below steps to reproduce the problem: create a project with a simple `gitlab-ci.yml` file: ``` build: stage: build tags: - ubuntu_amd64 script: - ping localhost ``` start a pipeline and cancel it. This should also terminate the ping command (but it doesn't) on the runner see if the process is still running ``` ps aux | grep ping gitlab-+ 19828 0.0 0.0 8656 1724 ? S 07:59 0:00 ping localhost ``` or just kill it with `killall ping` (use `sudo` if the runner is under another user) ## Proposal At the moment we are simply killing the process group with SIGKILL and then ignore the result. Instead of doing this we should allow the process to gracefully shutdown by first sending `SIGTERM` and after a specific timeout send `SIGKILL` to the process. This will help with the processes being killed properly. We already have this [implemented with the custom executor](https://docs.gitlab.com/runner/executors/custom.html#terminating-and-killing-executables) and should try and reuse the code to implement the same feature. ## Merge Requests 1. [x] [Extract process killer form custom executor](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1653) 1. [x] [Extract commander interface from custom executor](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1654) 1. [x] [Add Process groups to `process` pkg](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1743/) 1. [x] [Use the same termination commands on Windows](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/1797) - For windows on the shell executor, we pass [`taskkil`](https://gitlab.com/gitlab-org/gitlab-runner/blob/cc49cd18e2bd6b071f4c93bef5f0818fc5caf81c/helpers/process_group_windows.go#L16) while in the process package we just call [`process.Kill()`](https://gitlab.com/gitlab-org/gitlab-runner/blob/cc49cd18e2bd6b071f4c93bef5f0818fc5caf81c/executors/custom/process/killer_windows.go#L33) investigate which one is better or if we should use both. 1. [x] [Rename test file](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1784) 1. [x] [Send `SIGTERM` then `SIGKILL` for shell executor](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1770) *[Original merge request](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1551)*