Add concurrency limit to ProjectComplianceEvaluatorWorker

What does this MR do and why?

ComplianceManagement::ProjectComplianceEvaluatorWorker had no concurrency_limit. Its scheduler (FrameworkEvaluationSchedulerWorker) fans this worker out in batches with no throttling, and each job runs hundreds of queries against the primary database. On 2026-09-09/10 and again on 2026-10-08/09, that fan-out (observed around ~3,900 jobs in a single burst) exhausted the shared gitlabhq_production_sidekiq PgBouncer pool (capped at 69 connections).

Once that pool saturates, unrelated low-urgency workers start failing to get a DB connection and get deferred by Sidekiq's concurrency-limit middleware. That middleware defers every subsequent job for a worker once its buffer is non-empty, so a brief pool dip snowballs into a multi-hour, multi-million-job backlog. This is the mechanism behind INC-13984, INC-14022, and INC-14968, tracked in issue #517 (closed).

Screenshot_2026-10-11_at_9.23.08_PM

Full analysis in this html file Sidekiq_Pool_Saturation.html

This MR caps ProjectComplianceEvaluatorWorker at 20 concurrent jobs so it can no longer exhaust the pool by itself. The limit (20) is a conservative starting point in line with other DB-heavy, long-running workers (Releases::CreateEvidenceWorker, Import::DeletePlaceholderUserWorker), not a number backed by throughput sizing — worth validating with the infra team before/after merge.

This addresses the trigger side of the incident chain. The self-reinforcing "defer everything once non-empty" behavior in the concurrency-limit middleware itself is a separate, larger fix still tracked on issue #517 (closed).

References

Screenshots or screen recordings

N/A — backend worker change, no UI.

How to set up and validate locally

  1. In the Rails console: ComplianceManagement::ProjectComplianceEvaluatorWorker.get_concurrency_limit should return 20.
  2. Enqueue more than 20 ProjectComplianceEvaluatorWorker jobs at once and confirm only 20 run concurrently, with the rest buffered (Gitlab::SidekiqMiddleware::ConcurrencyLimit::ConcurrencyLimitService.queue_size('ComplianceManagement::ProjectComplianceEvaluatorWorker')).

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Shabini Rajadas

Merge request reports

Loading
Loading