RepositoryUpdateMirrorWorker is run too frequently and floods Sidekiq queues

This is what is flooding our Sidekiq queues:

  1. UpdateAllMirrorsWorker is run on the hour. There are about 13,000 mirrors.
  2. In batches of 200, it spawns RepositoryUpdateMirrorWorker to start launching jobs at random times between now and 30 minutes.
  3. This then adds RepositoryUpdateMirrorDispatchWorker to the queue with another 13,000 new entries.
  4. If there are retries, then more jobs are added.

In the graph below, you can see the effect of RepositoryUpdateMirrorWorker running followed by RepositoryUpdateMirrorDispatchWorker:

image

What's happening:

  1. RepositoryUpdateMirrorWorker: First, there are additional 13,000 jobs to update the DB.
  2. RepositoryUpdateMirrorDispatchWorker: Once those jobs are finished, another 13,000 jobs are used to launch git fetch.

Ideas for improvement:

  1. Back off the mirror time from 1 hour to something more (e.g. once a day--then we could spread 13,000 jobs across 24 hours)
  2. Only schedule a update if the project has not been updated in some time (e.g. if we just updated at 10:50 am, no sense in running at 11 am)
  3. Update the DB in batches instead of one worker per project

Other ideas?

/cc: @maratkalibek, @pcarranza, @yorickpeterse