ProjectDailyStatisticsWorker flooding GitLab.com queues
On GitLab.com, we have 20,501 jobs and growing due to ProjectDailyStatisticsWorker
, which was introduced in GitLab 11.9 via https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23596:
Part of the problem is that the same project may be adding to this queue (the counts include number of jobs for that project):
------------------------------
Top job counts with arguments:
------------------------------
["ProjectDailyStatisticsWorker", [278964]]: 1609
["ProjectDailyStatisticsWorker", [13083]]: 1296
["ProjectDailyStatisticsWorker", [10439512]]: 897
["PipelineProcessWorker", [58387792]]: 406
["ProjectDailyStatisticsWorker", [6053065]]: 159
We could spin up more Sidekiq workers here, but I worry that this job does a SQL update on the fetch_count
column and may run into lock contention when there are a lot of the same jobs running.
/cc: @nick.thomas, @jacopo-beschi
Edited by Stan Hu