Restarting the runner after changing the configuration creates another process
Summary
When i restart a GitLab runner in Docker after changing the configuration file, another runner process is started.
This causes a whole myriad of problems:
-
limit
andconcurrent
settings are ignored - Conflicts between Docker container instances occur frequently
Steps to reproduce
Start a runner through docker, then enter in the shell interactively.
root@ae17a1f6e7b4:/etc/gitlab-runner# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=1260 revision=a987417a version=12.2.0
Running in system-mode.
Registering runner... succeeded runner=tfKDMJSn
Merging configuration from template file "/etc/gitlab-runner/template-config.toml"
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
At this point, things are fine. There is one gitlab-runner process running and config.toml
contains one runner.
root@ae17a1f6e7b4:/etc/gitlab-runner# ps aux | grep "gitlab-runner run"
root 7 0.2 2.8 41508 27704 ? Ssl 08:08 0:00 gitlab-runner run --user=gitlab-runner --working-directory=/home/gitlab-runner
Running gitlab-runner restart
behaves as expected, there is still only one runner.
Next, i change the volume definition in config.toml
:
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "ae17a1f6e7b4"
limit = 1
url = "https://gitlab.mll"
token = "kxpymk3j5GfKyjx-imDk"
executor = "docker"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "docker:19.03.2-dind"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
- volumes = ["/cache"]
+ volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock", "/builds:/builds"]
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
And finally, i run gitlab-runner restart
root@ae17a1f6e7b4:/etc/gitlab-runner# gitlab-runner restart
Runtime platform arch=amd64 os=linux pid=1311 revision=a987417a version=12.2.0
Actual behavior
There are now two runner processes:
root@ae17a1f6e7b4:/etc/gitlab-runner# ps aux | grep "gitlab-runner run"
root 7 0.2 2.8 41508 27980 ? Ssl 08:08 0:00 gitlab-runner run --user=gitlab-runner --working-directory=/home/gitlab-runner
root 1333 7.6 2.7 41508 26780 ? Sl 08:16 0:00 /usr/lib/gitlab-runner/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user
Running gitlab-runner restart
does not change anything. I can get rid of the extra process by exiting the container and restarting it.
Expected behavior
There should still be only one runner process.
Environment description
I am using the gitlab/gitlab-runner:v12.2.0
container image, running on CoreOS using Server Version: 18.06.3-ce
.
Used GitLab Runner version
Version: 12.2.0
Git revision: a987417a
Git branch: 12-2-stable
GO version: go1.8.7
Built: 2019-08-22T13:06:00+0000
OS/Arch: linux/amd64
Root cause analysis
We never suggest using Docker containers as a replacement as VMs, if you want to change the config and restart the Runner it's best if you just start a new container, Docker containers don't allow for services supervisors like systemd. So you should consider a Docker container as a process and not a VM, if you want to restart the GitLab Runner process just start a new container not a new process. This will also cause problems because Docker expects the main process to be PID 1 and in this case it isn't so it can lead to other issues.
If you want to update the configuration we show an example of this for the Docker container in https://docs.gitlab.com/runner/register/index.html#docker which just runs a container 1 time, create config and then attach the config to another container.
I wonder if we should add this kind of information in https://docs.gitlab.com/runner/best_practice/index.html
I'm going to leave this issue open so we can take actin upon this, by either improving documentation or removing the system service commands like gitlab-runner restart
when inside of a Docker container.