Limit concurrency of standards adherence workers
What does this MR do and why?
A standards adherence refresh (Standards::GroupBaseWorker) enqueues one job per project in the group hierarchy for each of the six per-project check workers (for example Standards::Gitlab::DastWorker). These workers had no explicit concurrency limit.
This MR sets concurrency_limit to 30 per worker class on Standards::BaseWorker, so all six per-project workers inherit it. It is behind the boolean compliance_standards_worker_concurrency_limit flag (:instance actor). With the flag off, the lambda returns nil and the default limit applies, as before.
urgencystays:low. Switching to:throttledwould lower the default concurrency share from 25% to 15% even with the flag off, which the flag couldn't undo. Once the flag is on, buffered jobs count toward the:lowqueueing target, so switch to:throttledin the flag-removal MR.DastWorkerandSastWorkerare also enqueued from CI artifact uploads, so those jobs share the same limit.
Before enabling the flag (tracked in the rollout issue):
- Check the current per-worker concurrency and throughput for the six workers (
sidekiq_running_jobs,sidekiq_concurrency_limit_max_concurrent_jobs). Confirm 30 is well above the steady-state peak, so the limiter queue only absorbs refresh bursts. - If one large group's refresh delays other groups, consider
concurrency_limit_scope(:root_namespace, ...)as a follow-up.
References
Screenshots or screen recordings
Not applicable: no UI changes.
How to set up and validate locally
-
In the Rails console:
ComplianceManagement::Standards::Gitlab::DastWorker.get_concurrency_limit Feature.enable(:compliance_standards_worker_concurrency_limit) ComplianceManagement::Standards::Gitlab::DastWorker.get_concurrency_limit # => 30
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.