Improve the Distribution runner upgrade process
Updating the Distribution runners in https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/2434#note_1148513250 required a lot of manual babysitting:
- Since builds were running, the
apt install gitlab-runnerhung while the runner was waiting for the builds to finish. - New jobs weren't being picked up while this happened, so I made a copy of the
config.toml, moved the Prometheus listen address so the runner wouldn't conflict, and ran a new runner in a console while this one was finishing. - Eventually the
dpkgtimed out, leaving the package in a bad state:
$ sudo apt install gitlab-runner=15.4.0
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.
- I had to run
sudo dpkg --configure -ato correct the state. - Then
apt installcould proceed. - I then issued a
kill -SIGQUITto the new, temporary runner.
How do we deal with upgrades to shared runners again?
/cc: @tmaczukin, @ahanselka, @twk3