Monthly run of `ClearSharedRunnersMinutesWorker` causes burden on gitlab.com database servers
Once a month we have a sidekiq job ClearSharedRunnersMinutesWorker
which clears out the amount of shared runner minutes used. This job however causes a massive spike in database load, to the point where SREs have received pages about database performance relating to this. This means that it is also potentially impacting users during the period of execution
Last occurrence of this where SREs were paged is at gitlab-com/gl-infra/production#2779 (closed)
Ideally this job would be changed so that it is not as aggressive, or somehow overall less impactful on the system so we aren't causing alerts or impacting end users.
Proposal
ClearSharedRunnersMinutesWorker
triggers a job for every 100,000 namespaces. Rather than schedule them immediately we could use perform_in
to spread the batches across 3 hours period.