Runners are not de-registering properly
Summary
We had an incident after enabling ci_runner_limits #329438 (closed) where many GitLab.com users were unable to register new runners. Upon further investigation, we saw that there were a large number of runners which were not properly deregistering when they were finished.
We turned off the feature flag for now gitlab-com/gl-infra/production#4606 (closed) - but will need to determine why runners are remaining registered when they no longer exist.
Steps to reproduce
Enable feature flag: #329438 (closed)
Example Project
What is the current bug behavior?
See above for history.
This seems to have affected customers who had a high number of inactive runners.
What is the expected correct behavior?
Can we automatically remove stale/inactive runners? #15447 was requested a long time ago.
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)