Triage non-idempotent workers
At gitlab-org/gitlab!25202 (merged) we're introducing a idempotent cop check for the new idempotent worker attribute. It bypasses a series of workers that could be idempotent, but both need further triage and testing in order to receive a idempotent! call.
The workers that schedule the most duplicates (duplicates are jobs for the same worker, with the same arguments, while another job is already enqueued).
| json.class.keyword: Descending | Count | Sum of json.duration | Duration in hours | Issue |
|---|---|---|---|---|
| AuthorizedProjectsWorker | 1,169,309 | 1,631,372.94 | 453.1591506 | #280 (closed) |
| ProcessCommitWorker | 1,028,703 | 434,819.12 | 120.7830881 | #282 (closed) |
| WebHookWorker | 1,537,784 | 260,819.55 | 72.44987528 | gitlab-org/gitlab#218559 (closed) |
| ExpirePipelineCacheWorker | 439,763 | 88,945.36 | 24.70704417 | #284 (closed) |
| StageUpdateWorker | 747,685 | 87,157.24 | 24.21034417 | #285 (closed) |
| ProjectDailyStatisticsWorker | 479,511 | 70,250.28 | 19.51396556 | #256 (closed) |
| PipelineUpdateWorker | 308,843 | 32,321.07 | 8.978074444 | #287 (moved) |
| Namespaces::ScheduleAggregationWorker | 186,661 | 16,433.43 | 4.564840278 | #218 (closed) |
| UpdateHeadPipelineForMergeRequestWorker | 238,564 | 13,914.08 | 3.8650225 | #288 (closed) |
| ElasticIndexerWorker | 102,903 | 11,922.26 | 3.311739444 | #289 (closed) |
| ExpireJobCacheWorker | 57,341 | 11,189.24 | 3.108121667 | #290 (closed) |
| BuildHooksWorker | 27,700 | 11,183.11 | 3.106418611 | #291 (moved) |
| ProjectImportScheduleWorker | 16,119 | 8,245.43 | 2.290396944 | |
| UpdateAllMirrorsWorker | 2,526 | 6,537.76 | 1.816045 | |
| CleanupContainerRepositoryWorker | 446 | 5,442.65 | 1.5118475 | |
| RepositoryUpdateRemoteMirrorWorker | 231 | 5,171.90 | 1.436639722 | |
| ElasticCommitIndexerWorker | 11,387 | 4,259.01 | 1.1830575 | |
| ProjectCacheWorker | 83,969 | 4,166.36 | 1.157321667 | |
| MergeRequestMergeabilityCheckWorker | 2,674 | 4,124.13 | 1.145591944 | |
| ProjectServiceWorker | 71,781 | 3,944.42 | 1.095673333 | |
| ElasticIndexBulkCronWorker | 2,260 | 1,068.40 | 0.2967786111 | |
| AutoMergeProcessWorker | 4,179 | 857.239 | 0.2381219444 | |
| RepositoryUpdateMirrorWorker | 638 | 686.928 | 0.1908133333 | |
| ReactiveCachingWorker | 310 | 454.489 | 0.1262469444 | |
| PipelineHooksWorker | 2,215 | 415.77 | 0.1154916667 |
Source: https://log.gprd.gitlab.net/goto/a7fc2366e28f6321294d027ee376d92d
Edited by Luke Duncalfe