Skip to content

Make repository pull mirroring not depend on sidekiq queue sizes

Follow up from: gitlab-com/gl-infra/scalability#995

At the time of writing, the UpdateAllMirrorsWorker uses the queue size of the ProjectImportScheduleWorker before possibly rescheduling itself.

I believe this is done to spread the load out a bit more, making sure we don't schedule more ProjectImportScheduleWorkers and the following RepositoryUpdateMirrorWorkers than we can chew.

In gitlab-com/gl-infra&447 (closed) we're moving from having a queue per worker, to jobs for multiple workers being scheduled on a single queue.

This means that checking the queue size would no longer be valid for checking if the UpdateAllMirrorsWorker can continue scheduling jobs.

In the short term, we're keeping the ProjectImportScheduleWorker on its own queue so this mechanism stays correct. But ideally we'd figure out a better way to do this.