Sidekiq: `UpdateBuildQueueService`: Improve when many runners are configured for project
Some projects have many runners (even 10k), but majority of them are offline and not longer used.
GitLab currently does not have a mechanism to recycle these runners. This at least in one circumstance
creates a contention on UpdateBuildQueueService
. This service iterates all runners regardless if they are offline or online making to fire many queries and many Redis calls. This proved to be problematic to some of our customers.
This job also could use DB replicas...
MR improving that: !48379 (diffs)
Metrics
Unsure how much we can free, but we should definitely be able to reduce 95th percentiles for this frequent job.
https://log.gprd.gitlab.net/goto/0e6b6dd91fb1de79790108fe5474fa44
Edited by Kamil Trzciński