`UpdateAllMirrorsWorker` is not scheduling mirror updates fast enough
UpdateAllMirrorsWorker
is pretty much always running, but it cannot schedule fast enough to actually fill the capacity, because it's fighting against RepositoryUpdateMirrorWorker
s with a concurrency of 72 (4 workers, 2 sidekiq processes, 9 threads), which means that we're rarely actually using that max concurrency of 72.
It's currently scheduling approximately 3 jobs per second (per Kibana), which is not too bad. With 23k mirrors scheduled at a rate of 3 per second, we'd schedule all of them in a little over 2 hours, which matches the p99 of 3 hours on project update frequency we're currently seeing. Any outliers could just be getting punished for their error rate.
To get p99 down and get projects to actually update in accordance with their next_execution_ts
, we need to get the UpdateAllMirrorsWorker
schedule rate up so that we're always fully utilizing the available concurrency for RepositoryUpdateMirrorWorker
, and then see if we need to increase that concurrency.
/cc @tiagonbotelho