Skip to content

GitLab Next

Open
Created by Branislav Makan@bmakan

CI or CD process does not receive SIGTERM on termination

Summary

When the CI job is cancelled on the Web UI, the job process doesn't receive the SIGTERM signal. 10 minutes later SIGKILL is received and the process ends ungracefully.

Steps to reproduce

  1. Run a CI job.
  2. Cancel the running CI job.
.gitlab-ci.yml 
test:
  - echo "Will sleep"
  - sleep 5000

Actual behavior

The process running behind the CI job keeps running after the job termination.

Expected behavior

The job is supposed to receive a SIGTERM signal and terminate.

Relevant logs and/or screenshots

I set up a signal handler with logging in my python script.

Python Code Example 
import logging
import signal

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

formatter = logging.Formatter(LOG_FORMAT)
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
handler.setFormatter(formatter)

file_handler = logging.FileHandler('/tmp/bmakan.log')
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)

logger.addHandler(handler)
logger.addHandler(file_handler)

def sigterm_handler(signum, frame):
    logger.info(f'parent got shutdown with {signum}')
    global shutdown
    shutdown = True
    exit(1)

signal.signal(signal.SIGHUP, sigterm_handler)
signal.signal(signal.SIGINT, sigterm_handler)
signal.signal(signal.SIGQUIT, sigterm_handler)
signal.signal(signal.SIGILL, sigterm_handler)
signal.signal(signal.SIGTRAP, sigterm_handler)
signal.signal(signal.SIGABRT, sigterm_handler)
signal.signal(signal.SIGBUS, sigterm_handler)
signal.signal(signal.SIGFPE, sigterm_handler)
#signal.signal(signal.SIGKILL, sigterm_handler)
signal.signal(signal.SIGUSR1, sigterm_handler)
signal.signal(signal.SIGSEGV, sigterm_handler)
signal.signal(signal.SIGUSR2, sigterm_handler)
signal.signal(signal.SIGPIPE, sigterm_handler)
signal.signal(signal.SIGALRM, sigterm_handler)
signal.signal(signal.SIGTERM, sigterm_handler)

The logs are written into the stdout (accessingle via the GitLab web page) and into a separate log file.

  • When I cancel the job I don't see anything in the logs and the process keeps on running.
    • The jobs appears as cancelled on the web page.
  • When I run kill -SIGTERM <pid> I can see the logged message in my logfile and the process terminates.
Before pressing `Cancel` button 
root     13019     1 13019 13019  0 Jan14 ?        00:04:16   /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runn
root     29753 13019 29753 13019  0 12:32 ?        00:00:00     su -s /bin/bash gitlab-runner -c bash --login
gitlab-+ 29755 29753 29755 29755  0 12:32 ?        00:00:00       bash --login
gitlab-+ 29798 29755 29755 29755  0 12:32 ?        00:00:00         bash --login
gitlab-+ 29799 29798 29755 29755  0 12:32 ?        00:00:00           python python-script.py
Several seconds, few minutes after hitting the `Cancel` button 
gitlab-+ 29798     1 29755 29755  0 12:32 ?        00:00:00   bash --login
gitlab-+ 29799 29798 29755 29755  0 12:32 ?        00:00:00     python python-script.py

After 10 minutes, the SIGKILL is received at the process terminates ungracefully.

Environment description

  • GitLab FOSS: 13.7.4 (2f14978e280)
  • Executor: shell
config.toml contents 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "my-runner"
  url = "my-url"
  token = "my-token"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

Used GitLab Runner version

Version:      13.7.0
Git revision: 943fc252
Git branch:   13-7-stable
GO version:   go1.13.8
Built:        2020-12-21T13:47:06+0000
OS/Arch:      linux/amd64

Possible fixes

Edited by DarwinJS