Geo repository/download sync queue can grow large without any backpressure

Currently the RepositorySyncWorker and FileDownloadDispatchWorker run every minute, trying to fill at most 25 jobs every time. However, if the system is running slow, the queues could grow infinitely large. We don't actually check the current number of jobs in the queue.

In #3147 (closed), we are going to be adding shard-specific RepositorySyncWorker jobs. This may cause us to schedule even more jobs without considering how many jobs are in the queue currently.

I've moved the Geo downloads into their own queues so it's easier to track this: gitlab-org/gitlab-ee!3607

Questions:

  1. Do we want to prevent adding more jobs on the queue if we've reached a limit?
  2. Should we consider de-duplication of jobs here? For example, if we are already scheduling a repo/file sync, we shouldn't need to add another job for this.
Assignee Loading
Time tracking Loading