Add concurrency limit to ProjectComplianceEvaluatorWorker
What does this MR do and why?
ComplianceManagement::ProjectComplianceEvaluatorWorker had no concurrency_limit. Its scheduler (FrameworkEvaluationSchedulerWorker) fans this worker out in batches with no throttling, and each job runs hundreds of queries against the primary database. On 2026-09-09/10 and again on 2026-10-08/09, that fan-out (observed around ~3,900 jobs in a single burst) exhausted the shared gitlabhq_production_sidekiq PgBouncer pool (capped at 69 connections).
Once that pool saturates, unrelated low-urgency workers start failing to get a DB connection and get deferred by Sidekiq's concurrency-limit middleware. That middleware defers every subsequent job for a worker once its buffer is non-empty, so a brief pool dip snowballs into a multi-hour, multi-million-job backlog. This is the mechanism behind INC-13984, INC-14022, and INC-14968, tracked in issue #517 (closed).
Full analysis in this html file Sidekiq_Pool_Saturation.html
This MR caps ProjectComplianceEvaluatorWorker at 20 concurrent jobs so it can no longer exhaust the pool by itself. The limit (20) is a conservative starting point in line with other DB-heavy, long-running workers (Releases::CreateEvidenceWorker, Import::DeletePlaceholderUserWorker), not a number backed by throughput sizing — worth validating with the infra team before/after merge.
This addresses the trigger side of the incident chain. The self-reinforcing "defer everything once non-empty" behavior in the concurrency-limit middleware itself is a separate, larger fix still tracked on issue #517 (closed).
References
- gitlab-com/gl-infra/tenant-scale/tenant-services/team#517 (closed)
- https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23138
Screenshots or screen recordings
N/A — backend worker change, no UI.
How to set up and validate locally
- In the Rails console:
ComplianceManagement::ProjectComplianceEvaluatorWorker.get_concurrency_limitshould return20. - Enqueue more than 20
ProjectComplianceEvaluatorWorkerjobs at once and confirm only 20 run concurrently, with the rest buffered (Gitlab::SidekiqMiddleware::ConcurrencyLimit::ConcurrencyLimitService.queue_size('ComplianceManagement::ProjectComplianceEvaluatorWorker')).
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
