Limit concurrency of standards adherence workers

What does this MR do and why?

A standards adherence refresh (Standards::GroupBaseWorker) enqueues one job per project in the group hierarchy for each of the six per-project check workers (for example Standards::Gitlab::DastWorker). These workers had no explicit concurrency limit.

This MR sets concurrency_limit to 30 per worker class on Standards::BaseWorker, so all six per-project workers inherit it. It is behind the boolean compliance_standards_worker_concurrency_limit flag (:instance actor). With the flag off, the lambda returns nil and the default limit applies, as before.

  • urgency stays :low. Switching to :throttled would lower the default concurrency share from 25% to 15% even with the flag off, which the flag couldn't undo. Once the flag is on, buffered jobs count toward the :low queueing target, so switch to :throttled in the flag-removal MR.
  • DastWorker and SastWorker are also enqueued from CI artifact uploads, so those jobs share the same limit.

Before enabling the flag (tracked in the rollout issue):

  • Check the current per-worker concurrency and throughput for the six workers (sidekiq_running_jobs, sidekiq_concurrency_limit_max_concurrent_jobs). Confirm 30 is well above the steady-state peak, so the limiter queue only absorbs refresh bursts.
  • If one large group's refresh delays other groups, consider concurrency_limit_scope(:root_namespace, ...) as a follow-up.

References

Screenshots or screen recordings

Not applicable: no UI changes.

How to set up and validate locally

  1. In the Rails console:

    ComplianceManagement::Standards::Gitlab::DastWorker.get_concurrency_limit
    Feature.enable(:compliance_standards_worker_concurrency_limit)
    ComplianceManagement::Standards::Gitlab::DastWorker.get_concurrency_limit # => 30

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports

Loading
Loading