Failing to upgrade to 13.2.0-rc1

While deploying 13.2.0-rc to the private runner manager fleet in GitLab.com as part of #26450 (closed) I started seeing the following issues when trying to start the service:

● gitlab-runner.service - GitLab Runner
   Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/gitlab-runner.service.d
           └─exec_start.conf, kill.conf, limits.conf
   Active: activating (auto-restart) (Result: exit-code) since Mon 2020-07-13 07:19:20 UTC; 1min 40s ago
  Process: 9712 ExecStart=/usr/lib/gitlab-runner/gitlab-runner --log-format json run --working-directory /home/gitlab-runner --config /etc/gitlab
 Main PID: 9712 (code=exited, status=203/EXEC)

Jul 13 07:19:20 private-runners-manager-3 systemd[1]: gitlab-runner.service: Unit entered failed state.
Jul 13 07:19:20 private-runners-manager-3 systemd[1]: gitlab-runner.service: Failed with result 'exit-code'.

Notice the (code=exited, status=203/EXEC) error.

If we check the service installed with cat /etc/systemd/system/gitlab-runner.service

# cat /etc/systemd/system/gitlab-runner.service
[Unit]
Description=GitLab Runner
After=syslog.target network.target
ConditionFileIsExecutable=/usr/bin/gitlab-runner

[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/bin/gitlab-runner "run" "--working-directory" "/home/gitlab-runner" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--syslog" "--user" "gitlab-runner"





Restart=always
RestartSec=120

[Install]
WantedBy=multi-user.target

Whilst on a correct runner manager the service file is as follows:

root@private-runners-manager-4.gitlab.com:~# cat /etc/systemd/system/gitlab-runner.service
[Unit]
Description=GitLab Runner
After=syslog.target network.target
ConditionFileIsExecutable=/usr/lib/gitlab-runner/gitlab-runner

[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/lib/gitlab-runner/gitlab-runner "run" "--working-directory" "/home/gitlab-runner" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--syslog" "--user" "gitlab-runner"





Restart=always
RestartSec=120

[Install]
WantedBy=multi-user.target

Notice that the ExecStart changed from /usr/lib/gitlab-runner/gitlab-runner to /usr/bin/gitlab-runner. Which makes sense becuase we changed the Runner location from /usr/lib/gitlab-runner/gitlab-runner to /usr/bin/gitlab-runner in !2176 (merged).

Now if we look at where the gitlab-runner binary is installed we see the following:

# The machine that is failing to start service
which gitlab-runner
/usr/bin/gitlab-runner
root@private-runners-manager-3.gitlab.com:~# ls -la /usr/bin/gitlab-runner
-rwxr-xr-x 1 root root 35295232 Jul  9 07:04 /usr/bin/gitlab-runner

# The machine that starts the servie
which gitlab-runner
/usr/bin/gitlab-runner
/usr/bin/gitlab-runner
lrwxrwxrwx 1 root root 34 Jul  1 06:56 /usr/bin/gitlab-runner -> ../lib/gitlab-runner/gitlab-runner

Just for testing purposes on the faulty machine, I ran the following and it fixed the service automaticaaly on the faulty machine.

# Moving the binary to the path pre 13.2.0-rc2
mv /usr/bin/gitlab-runner /usr/lib/gitlab-runner/gitlab-runner
# Making sure gitlab-runner is not available anymore
gitlab-runner
-bash: /usr/bin/gitlab-runner: No such file or directory
# Create symlink similar to what the machine has
ln -s /usr/lib/gitlab-runner/gitlab-runner /usr/bin/gitlab-runner
which gitlab-runner
/usr/bin/gitlab-runner
ls -la /usr/bin/gitlab-runner
lrwxrwxrwx 1 root root 36 Jul 13 07:30 /usr/bin/gitlab-runner -> /usr/lib/gitlab-runner/gitlab-runner
# Check if service is running
root@private-runners-manager-3.gitlab.com:~# gitlab-runner status
Runtime platform                                    arch=amd64 os=linux pid=12706 revision=4acba09b version=13.2.0-rc1
gitlab-runner: Service is running!

For some reason our service detention requires a symlink to work as expected 🤔

Proposal

As discussed in #26582 (comment 378246140) this can be considered a breaking change. We will revert !2176 (merged) and then in %13.3 reintroduce the change with where /usr/lib/gitlab-runner/gitlab-runner is a symlink to /usr/bin/gitlab-runner.