Improve the Distribution runner upgrade process
Updating the Distribution runners in https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/2434#note_1148513250 required a lot of manual babysitting:
- Since builds were running, the
apt install gitlab-runner
hung while the runner was waiting for the builds to finish. - New jobs weren't being picked up while this happened, so I made a copy of the
config.toml
, moved the Prometheus listen address so the runner wouldn't conflict, and ran a new runner in a console while this one was finishing. - Eventually the
dpkg
timed out, leaving the package in a bad state:
$ sudo apt install gitlab-runner=15.4.0
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.
- I had to run
sudo dpkg --configure -a
to correct the state. - Then
apt install
could proceed. - I then issued a
kill -SIGQUIT
to the new, temporary runner.
How do we deal with upgrades to shared runners again?
/cc: @tmaczukin, @ahanselka, @twk3