Skip to content

Improve the Distribution runner upgrade process

Updating the Distribution runners in https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/2434#note_1148513250 required a lot of manual babysitting:

  1. Since builds were running, the apt install gitlab-runner hung while the runner was waiting for the builds to finish.
  2. New jobs weren't being picked up while this happened, so I made a copy of the config.toml, moved the Prometheus listen address so the runner wouldn't conflict, and ran a new runner in a console while this one was finishing.
  3. Eventually the dpkg timed out, leaving the package in a bad state:
$ sudo apt install gitlab-runner=15.4.0
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.
  1. I had to run sudo dpkg --configure -a to correct the state.
  2. Then apt install could proceed.
  3. I then issued a kill -SIGQUIT to the new, temporary runner.

How do we deal with upgrades to shared runners again?

/cc: @tmaczukin, @ahanselka, @twk3