Graceful shutdown & OS Respawning
I turn some of my build servers off at 7pm, and the graceful shutdown left me quite confused - although one could argue it's not exactly a bug.
In line with the documentation I issue a sudo killall -SIGQUIT gitlab-ci-multi-runner
then monitored the output of sudo gitlab-runner status
but on Ubuntu 14.04, the status kept reporting the service was running even after all builds had finished.
This was because, in the default configuration, the OS restarts the runner as soon as it stops:
Oct 12 21:43:42 ip-10-74-182-78 gitlab-runner[3161]: WARN[0170] Requested quit, waiting for builds to finish builds=1
Oct 12 21:44:36 ip-10-74-182-78 gitlab-runner[3161]: INFO[0225] Build succeeded build=5048840 project=1834016 runner=2fa37e50
Oct 12 21:44:39 ip-10-74-182-78 gitlab-runner[3161]: INFO[0227] All workers stopped. Can exit now builds=0
Oct 12 21:44:39 ip-10-74-182-78 kernel: [ 373.721568] init: gitlab-runner main process ended, respawning
Oct 12 21:44:39 ip-10-74-182-78 gitlab-runner[3824]: INFO[0000] Starting multi-runner from /etc/gitlab-runner/config.toml ... builds=0
Oct 12 21:44:39 ip-10-74-182-78 gitlab-runner[3824]: INFO[0000] Running in system-mode.
Oct 12 21:44:39 ip-10-74-182-78 gitlab-runner[3824]: INFO[0000]
Oct 12 21:44:39 ip-10-74-182-78 gitlab-runner[3824]: INFO[0000] Configuration loaded builds=0
This doesn't appear to be an issue on Ubuntu 16.04 - presumably because of the upstart/systemd change over.
So what else could we do?
- As long as killall is the suggested way to issue these commands, there's nowhere much to write a warning message (apart from a log file the user might not know about)
- Could change the default service config file not to respawn the runner. I don't know how crashy it is, so I don't know if that's a good idea.
- Could change the service config file on receipt of sigquit. This is a terrible idea.
- Could add a
gitlab-runner graceful-shutdown
command that just sort of sorts things out (and warns if this is going to happen) - Could add acknowledgement of the graceful shutdown to
gitlab-runner status
- Could improve the documentation everywhere a sigquit is mentioned.
Detailed reproduction steps:
Launched an Ubuntu 14.04 EC2 instance:
aws ec2 request-spot-instances --spot-price 0.05 --instance-count 1 --launch-specification "{\"ImageId\":\"ami-7842030f\",\"InstanceType\":\"m3.medium\",\"KeyName\":\"mtandy home i7 pc\",\"IamInstanceProfile\":{\"Name\":\"s3access\"},\"SecurityGroupIds\":[\"sg-2e039e59\"],\"BlockDeviceMappings\":[{\"VirtualName\":\"ephemeral0\",\"DeviceName\":\"/dev/sdb\"}]}"
Installed runner like so:
sudo apt-get update; sudo apt-get install -y docker.io
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash
sudo apt-get install gitlab-ci-multi-runner
Paired it to a trivial project using the docker runner.
While a build was in progress, issued this command:
sudo killall -SIGQUIT gitlab-ci-multi-runner; for i in `seq 1 180`; do gitlab-runner status || break; sleep 10; done