## Summary The `low_urgency_cpu_bound` Sidekiq queue is experiencing significant congestion due to workers taking longer than expected to complete. ## Problem Statement Sidekiq processes are a limited resource. When workers in the `low_urgency_cpu_bound` queue exceed their expected execution times, they create a backlog that affects all other jobs in the same queue. This leads to degraded user experience across multiple GitLab features. **User Impact Example:** The Pipeline Authoring team has observed that `ExternalServiceReactiveCachingWorker` jobs frequently wait 10-30 seconds in the queue before execution begins, preventing `Ci::ListConfigVariablesService` from returning results to users in a timely manner. https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21198+ ## Data Analysis Analysis of production logs over the last 7 days (Aug 17-23, 2026) reveals the following slow workers: ### Jobs with duration > 10 seconds (1,020,000 total jobs) https://log.gprd.gitlab.net/app/r/s/HY0Ly | Worker | Count | Percentage | |--------|-------|------------| | `ReactiveCaching::LowUrgencyWorker` | 417,998 | 41.0% | | `PipelineHooksWorker` | 197,608 | 19.4% | | `AutoMergeProcessWorker` | 61,796 | 6.1% | | `Ci::PipelineFinishedWorker` | 55,954 | 5.5% | | `Ci::CreateDownstreamPipelineWorker` | 48,829 | 4.8% | | `MailScheduler::NotificationServiceWorker` | 37,477 | 3.7% | | `Security::Scans::IngestReportsWorker` | 32,161 | 3.2% | | `Sbom::IngestReportsWorker` | 22,016 | 2.2% | | `Ci::TrackFailedBuildWorker` | 21,586 | 2.1% | | `AuthorizedProjectUpdate::UserRefreshFromReplicaWorker` | 20,505 | 2.0% | | Other | 104,070 | 10.2% | ### Jobs with duration > 30 seconds (771,485 total jobs) https://log.gprd.gitlab.net/app/r/s/tYviC | Worker | Count | Percentage | |--------|-------|------------| | `ReactiveCaching::LowUrgencyWorker` | 383,730 | 49.7% | | `PipelineHooksWorker` | 120,107 | 15.6% | | `Ci::CreateDownstreamPipelineWorker` | 67,477 | 8.7% | | `Sbom::IngestReportsWorker` | 44,586 | 5.8% | | `Ci::Observability::ExportWorker` | 25,835 | 3.3% | | `MailScheduler::NotificationServiceWorker` | 22,159 | 2.9% | | `MergeRequests::NotifyApproversWorker` | 12,236 | 1.6% | | `Sbom::BuildDependencyGraphWorker` | 10,340 | 1.3% | | `MergeTrains::RefreshWorker` | 9,459 | 1.2% | | `Projects::ImportExport::CreateRelationExportsWorker` | 9,406 | 1.2% | | Other | 66,150 | 8.6% | ## Key Observations 1. **`ReactiveCaching::LowUrgencyWorker`** is now the single largest contributor in both buckets — 41.0% of jobs over 10s and 49.7% of jobs over 30s. It accounts for nearly half of the worst offenders, so reducing its execution time is the highest-leverage change for the queue. 2. **`PipelineHooksWorker`** is consistently the second-largest contributor (19.4% at >10s, 15.6% at >30s), a much larger share than in prior analysis. It warrants dedicated investigation. 3. **`Ci::CreateDownstreamPipelineWorker`** remains a steady top-5 contributor in both buckets (4.8% / 8.7%), with its share growing at the >30s threshold. 4. **`MailScheduler::NotificationServiceWorker`**, previously a top-3 offender, has dropped substantially (3.7% / 2.9%), suggesting earlier concerns have eased. 5. **Security and SBOM workers** (`Security::Scans::IngestReportsWorker`, `Sbom::IngestReportsWorker`, `Sbom::BuildDependencyGraphWorker`) together remain a meaningful chunk of slow jobs across both thresholds. ## Proposed Actions Each child issue should investigate the specific worker and determine: - Why jobs are taking longer than expected - Potential optimizations to reduce execution time - Whether timeouts or circuit breakers should be implemented (e.g. https://gitlab.com/gitlab-org/gitlab/-/work_items/588313#note_3092592721) For `ReactiveCaching::LowUrgencyWorker` specifically, we need to: - Identify all classes using reactive caching that run through this worker - Break down execution times by the underlying class being executed - Determine which classes are the biggest contributors to queue congestion - Consider whether high-impact classes like `Ci::ListConfigVariablesService` need dedicated optimization or alternative approaches ### Breakdown of `ReactiveCaching::LowUrgencyWorker` This worker is a generic wrapper that executes various classes using the [Reactive Caching pattern](https://docs.gitlab.com/development/reactive_caching/). The tables below break its slow jobs down two ways: by the underlying class being executed (`related_class`) and by the request that originated the caching (`root_caller_id`). #### By related class **Jobs with duration > 10 seconds** https://log.gprd.gitlab.net/app/r/s/nV0mA | Class | Count | Percentage | |-------|-------|------------| | `Security::MergeRequestSecurityReportGenerationService` | 740,247 | 72.6% | | `MergeRequest` | 217,269 | 21.3% | | `Ci::ListConfigVariablesService` | 24,195 | 2.4% | | `Ci::PipelineCreation::FindPipelineInputsService` | 23,139 | 2.3% | | `Group` | 15,150 | 1.5% | **Jobs with duration > 30 seconds** https://log.gprd.gitlab.net/app/r/s/n2Lnr | Class | Count | Percentage | |-------|-------|------------| | `Security::MergeRequestSecurityReportGenerationService` | 306,071 | 77.2% | | `MergeRequest` | 85,903 | 21.7% | | `Ci::ListConfigVariablesService` | 3,737 | 0.9% | | `Ci::PipelineCreation::FindPipelineInputsService` | 430 | 0.1% | | `Group` | 266 | < 0.1% | <details> <summary>Click to expand</summary> #### By root caller **Jobs with duration > 10 seconds** https://log.gprd.gitlab.net/app/r/s/nV0mA | Root caller | Count | Percentage | |-------------|-------|------------| | `GraphqlController#execute` | 559,281 | 54.8% | | `GraphqlController#execute:FindingReportsComparer` | 178,676 | 17.5% | | `Projects::MergeRequests::ContentController#cached_widget` | 103,488 | 10.1% | | `Projects::MergeRequestsController#test_reports` | 64,895 | 6.4% | | `Cronjob` | 20,685 | 2.0% | | `Projects::MergeRequestsController#codequality_reports` | 18,350 | 1.8% | | `Projects::MergeRequestsController#license_scanning_reports_collapsed` | 13,169 | 1.3% | | `GraphqlController#execute:getMrSastReport` | 12,756 | 1.3% | | `GraphqlController#execute:ciPipelineCreationInputs` | 9,818 | 1.0% | | `GraphqlController#execute:getMRCodequalityAndSecurityReportsEE` | 6,235 | 0.6% | | Other | 32,646 | 3.2% | **Jobs with duration > 30 seconds** https://log.gprd.gitlab.net/app/r/s/n2Lnr | Root caller | Count | Percentage | |-------------|-------|------------| | `GraphqlController#execute` | 207,686 | 52.4% | | `GraphqlController#execute:FindingReportsComparer` | 88,315 | 22.3% | | `Projects::MergeRequests::ContentController#cached_widget` | 36,936 | 9.3% | | `Projects::MergeRequestsController#test_reports` | 27,971 | 7.1% | | `Projects::MergeRequestsController#codequality_reports` | 14,416 | 3.6% | | `Cronjob` | 6,448 | 1.6% | | `GraphqlController#execute:getMrSastReport` | 6,321 | 1.6% | | `Projects::MergeRequestsController#license_scanning_reports_collapsed` | 3,335 | 0.8% | | `GraphqlController#execute:getMRCodequalityAndSecurityReportsEE` | 3,151 | 0.8% | | `GraphqlController#execute:ciConfigVariables` | 649 | 0.2% | | Other | 1,178 | 0.3% | </details> ## Success Criteria - Reduce the number of jobs exceeding 30 seconds - Improve queue latency for the queue