Failing to upgrade to 13.2.0-rc1
While deploying 13.2.0-rc
to the private runner manager fleet in GitLab.com as part of #26450 (closed) I started seeing the following issues when trying to start the service:
● gitlab-runner.service - GitLab Runner
Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/gitlab-runner.service.d
└─exec_start.conf, kill.conf, limits.conf
Active: activating (auto-restart) (Result: exit-code) since Mon 2020-07-13 07:19:20 UTC; 1min 40s ago
Process: 9712 ExecStart=/usr/lib/gitlab-runner/gitlab-runner --log-format json run --working-directory /home/gitlab-runner --config /etc/gitlab
Main PID: 9712 (code=exited, status=203/EXEC)
Jul 13 07:19:20 private-runners-manager-3 systemd[1]: gitlab-runner.service: Unit entered failed state.
Jul 13 07:19:20 private-runners-manager-3 systemd[1]: gitlab-runner.service: Failed with result 'exit-code'.
Notice the (code=exited, status=203/EXEC)
error.
If we check the service installed with cat /etc/systemd/system/gitlab-runner.service
# cat /etc/systemd/system/gitlab-runner.service
[Unit]
Description=GitLab Runner
After=syslog.target network.target
ConditionFileIsExecutable=/usr/bin/gitlab-runner
[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/bin/gitlab-runner "run" "--working-directory" "/home/gitlab-runner" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--syslog" "--user" "gitlab-runner"
Restart=always
RestartSec=120
[Install]
WantedBy=multi-user.target
Whilst on a correct runner manager the service file is as follows:
root@private-runners-manager-4.gitlab.com:~# cat /etc/systemd/system/gitlab-runner.service
[Unit]
Description=GitLab Runner
After=syslog.target network.target
ConditionFileIsExecutable=/usr/lib/gitlab-runner/gitlab-runner
[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/lib/gitlab-runner/gitlab-runner "run" "--working-directory" "/home/gitlab-runner" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--syslog" "--user" "gitlab-runner"
Restart=always
RestartSec=120
[Install]
WantedBy=multi-user.target
Notice that the ExecStart
changed from /usr/lib/gitlab-runner/gitlab-runner
to /usr/bin/gitlab-runner
. Which makes sense becuase we changed the Runner location from /usr/lib/gitlab-runner/gitlab-runner
to /usr/bin/gitlab-runner
in !2176 (merged).
Now if we look at where the gitlab-runner
binary is installed we see the following:
# The machine that is failing to start service
which gitlab-runner
/usr/bin/gitlab-runner
root@private-runners-manager-3.gitlab.com:~# ls -la /usr/bin/gitlab-runner
-rwxr-xr-x 1 root root 35295232 Jul 9 07:04 /usr/bin/gitlab-runner
# The machine that starts the servie
which gitlab-runner
/usr/bin/gitlab-runner
/usr/bin/gitlab-runner
lrwxrwxrwx 1 root root 34 Jul 1 06:56 /usr/bin/gitlab-runner -> ../lib/gitlab-runner/gitlab-runner
Just for testing purposes on the faulty machine, I ran the following and it fixed the service automaticaaly on the faulty machine.
# Moving the binary to the path pre 13.2.0-rc2
mv /usr/bin/gitlab-runner /usr/lib/gitlab-runner/gitlab-runner
# Making sure gitlab-runner is not available anymore
gitlab-runner
-bash: /usr/bin/gitlab-runner: No such file or directory
# Create symlink similar to what the machine has
ln -s /usr/lib/gitlab-runner/gitlab-runner /usr/bin/gitlab-runner
which gitlab-runner
/usr/bin/gitlab-runner
ls -la /usr/bin/gitlab-runner
lrwxrwxrwx 1 root root 36 Jul 13 07:30 /usr/bin/gitlab-runner -> /usr/lib/gitlab-runner/gitlab-runner
# Check if service is running
root@private-runners-manager-3.gitlab.com:~# gitlab-runner status
Runtime platform arch=amd64 os=linux pid=12706 revision=4acba09b version=13.2.0-rc1
gitlab-runner: Service is running!
For some reason our service detention requires a symlink to work as expected
Proposal
As discussed in #26582 (comment 378246140) this can be considered a breaking change. We will revert !2176 (merged) and then in %13.3 reintroduce the change with where /usr/lib/gitlab-runner/gitlab-runner
is a symlink to /usr/bin/gitlab-runner
.