## Status update of 2024-01-17 Several enhancements were implemented to improve the handling of the `SIGTERM` signal in both ~"executor::docker" (!4446) and ~"executor::kubernetes" (!4443, !4485). However, no fix has been implemented yet for ~"executor::shell". ## Summary <!-- Briefly summarize the bug --> When the CI job is cancelled on the Web UI, the job process doesn't receive the `SIGTERM` signal. 10 minutes later `SIGKILL` is received and the process ends ungracefully. ## Steps to reproduce <!-- What do you need to do to reproduce the bug? Please include job definitions or git repository structure if relevant --> 1. Run a CI job. 1. Cancel the running CI job. <!-- Please add the definition of the job from `.gitlab-ci.yml` that is failing inside of the code blocks (```) below. --> <details> <summary> .gitlab-ci.yml </summary> ```yml test: - echo "Will sleep" - sleep 5000 ``` </details> ## Actual behavior <!-- What actually happens --> The process running behind the CI job keeps running after the job termination. ## Expected behavior <!-- What you should see instead --> The job is supposed to receive a `SIGTERM` signal and terminate. ## Relevant logs and/or screenshots <!-- Paste the job logs inside of the code blocks (```) below so it would be easier to read. --> I set up a signal handler with logging in my python script. <details> <summary>Python Code Example</summary> ```python import logging import signal logger = logging.getLogger() logger.setLevel(logging.DEBUG) formatter = logging.Formatter(LOG_FORMAT) handler = logging.StreamHandler(sys.stdout) handler.setLevel(logging.DEBUG) handler.setFormatter(formatter) file_handler = logging.FileHandler('/tmp/bmakan.log') file_handler.setLevel(logging.DEBUG) file_handler.setFormatter(formatter) logger.addHandler(handler) logger.addHandler(file_handler) def sigterm_handler(signum, frame): logger.info(f'parent got shutdown with {signum}') global shutdown shutdown = True exit(1) signal.signal(signal.SIGHUP, sigterm_handler) signal.signal(signal.SIGINT, sigterm_handler) signal.signal(signal.SIGQUIT, sigterm_handler) signal.signal(signal.SIGILL, sigterm_handler) signal.signal(signal.SIGTRAP, sigterm_handler) signal.signal(signal.SIGABRT, sigterm_handler) signal.signal(signal.SIGBUS, sigterm_handler) signal.signal(signal.SIGFPE, sigterm_handler) #signal.signal(signal.SIGKILL, sigterm_handler) signal.signal(signal.SIGUSR1, sigterm_handler) signal.signal(signal.SIGSEGV, sigterm_handler) signal.signal(signal.SIGUSR2, sigterm_handler) signal.signal(signal.SIGPIPE, sigterm_handler) signal.signal(signal.SIGALRM, sigterm_handler) signal.signal(signal.SIGTERM, sigterm_handler) ``` </details> The logs are written into the stdout (accessingle via the GitLab web page) and into a separate log file. * When I cancel the job I don't see anything in the logs and the process keeps on running. * The jobs appears as cancelled on the web page. * When I run `kill -SIGTERM <pid>` I can see the logged message in my logfile and the process terminates. <details> <summary>Before pressing `Cancel` button</summary> ```shell root 13019 1 13019 13019 0 Jan14 ? 00:04:16 /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runn root 29753 13019 29753 13019 0 12:32 ? 00:00:00 su -s /bin/bash gitlab-runner -c bash --login gitlab-+ 29755 29753 29755 29755 0 12:32 ? 00:00:00 bash --login gitlab-+ 29798 29755 29755 29755 0 12:32 ? 00:00:00 bash --login gitlab-+ 29799 29798 29755 29755 0 12:32 ? 00:00:00 python python-script.py ``` </details> <details> <summary>Several seconds, few minutes after hitting the `Cancel` button</summary> ```shell gitlab-+ 29798 1 29755 29755 0 12:32 ? 00:00:00 bash --login gitlab-+ 29799 29798 29755 29755 0 12:32 ? 00:00:00 python python-script.py ``` </details> After 10 minutes, the `SIGKILL` is received at the process terminates ungracefully. ## Environment description <!-- Are you using shared Runners on GitLab.com? Or is it a custom installation? Which executors are used? Please also provide the versions of related tools like `docker info` if you are using the Docker executor. --> * GitLab FOSS: 13.7.4 (2f14978e280) * Executor: shell <!-- Please add the contents of `config.toml` inside of the code blocks (```) below, remember to remove any secret tokens! --> <details> <summary> config.toml contents </summary> ```toml concurrent = 1 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "my-runner" url = "my-url" token = "my-token" executor = "shell" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] ``` </details> ### Used GitLab Runner version <!-- Please run and paste the output of `gitlab-runner --version`. If you are using a Runner where you don't have access to, please paste at least the first lines the from build log, like: --> ``` Version: 13.7.0 Git revision: 943fc252 Git branch: 13-7-stable GO version: go1.13.8 Built: 2020-12-21T13:47:06+0000 OS/Arch: linux/amd64 ``` ## Possible fixes <!-- (If you can, link to the line of code that might be responsible for the problem) --->