Runner doesn't come back online after restarting

Summary

We have observed unreliable connection for some time when trying to restart runner after updating config.toml file.

Steps to reproduce

1. Install runner in docker container and register to a test project.
2. Update runner's policy in config.toml
3. Restart container and wait for runner to be back online.

Example Project

What is the current bug behavior?

Runners at times just wouldn't come back online.

What is the expected correct behavior?

Should come back online within 60 seconds.

Relevant logs and/or screenshots

See failure issues:

• https://gitlab.com/gitlab-org/gitlab/-/issues/461290
• https://gitlab.com/gitlab-org/gitlab/-/issues/498988
• https://gitlab.com/gitlab-org/gitlab/-/issues/504929
• https://gitlab.com/gitlab-org/gitlab/-/issues/504943

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes