Scope SyncAnyMergeRequestApprovalRulesWorker per top-level group
What does this MR do and why?
Makes Security::ScanResultPolicies::SyncAnyMergeRequestApprovalRulesWorker the first worker to use the per-top-level-group concurrency limit from !264227 (merged). Each top-level group can use at most half of the worker's limit of 200, so a burst from one group no longer defers every other group's approval-rule syncs.
This worker is the best first candidate. From GitLab.com Sidekiq logs over the last 7 days:
- It was deferred 47,593 times, every day, across 121 top-level groups. It ran at up to 199 of its 200 slots, with up to 3,663 jobs waiting.
- Three top-level groups caused 85% of those deferrals (43%, 25% and 17%), so a cap frees real capacity for the other groups.
- 90% of its deferred jobs had a top-level group in their context, so the cap applies to most of its traffic.
SyncMergeRequestApprovalsWorker, the worker from the original incident, wasn't deferred at all in the same window, so a scope there would have nothing to cap yet.
Rollout: the scope has its own gitlab_com_derisk flag, sync_any_merge_request_approval_rules_worker_concurrency_limit_scope, checked against :instance, so this worker rolls out separately from the other adopters. It is off by default. The scope only applies once that flag and sidekiq_concurrency_limit_scope are both enabled (#636157). The only other workers that declare a scope are CreatePipelineWorker and MergeRequests::CreatePipelineWorker. Their per-user limit: comes from GITLAB_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER, which defaults to 0, so they stay unscoped unless that variable is set. To turn the scope off for this worker alone, disable its flag, or enable disable_sidekiq_concurrency_limit_scope_Security::ScanResultPolicies::SyncAnyMergeRequestApprovalRulesWorker.
This worker is high-volume, so its flag stays off on GitLab.com until #637499 lands. This MR only wires the scope in; a low-volume Policy Management worker goes first.
Builds on !269153 (merged) (merged), which makes the sidekiq_concurrency_limit_queue_jobs gauge exact for scoped workers. We need that gauge to judge this rollout.
References
How to set up and validate locally
In bin/rails runner or a console, with both flags on:
require 'sidekiq/testing'
Feature.enable(:sidekiq_concurrency_limit_scope)
Feature.enable(:sync_any_merge_request_approval_rules_worker_concurrency_limit_scope)
worker = Security::ScanResultPolicies::SyncAnyMergeRequestApprovalRulesWorker
scope = Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(worker)
scope.limit_for(200) # => 100
projects = Group.where(parent_id: nil).first(200).select { |g| g.all_projects.exists? }.first(2).map { |g| g.all_projects.first }
Sidekiq::Testing.fake! do
projects.each_with_index { |p, i| Gitlab::ApplicationContext.with_context(project: p) { worker.perform_async(-1 - i) } }
worker.perform_async(-99)
worker.jobs.map { |job| scope.key_for(job) } # => ["root_namespace:toolbox", "root_namespace:gitlab-org", nil]
worker.jobs.clear
end
Feature.disable(:sidekiq_concurrency_limit_scope)
Feature.disable(:sync_any_merge_request_approval_rules_worker_concurrency_limit_scope)With either flag off, Scope.for(worker) returns nil.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Reviewable LOC
4 lines of application code, 20 lines of specs and the feature flag definition.
Context for LLM agents
max_percentage: 0.5is a deliberately generous start. Tighten it once the per-scope metrics are in.- The worker uses
data_consistency :always. The scope only defers jobs and never adds any, so the load on the primary database stays bounded by the existing limit of 200.
Owed elsewhere
- Enable
sidekiq_concurrency_limit_scopeandsync_any_merge_request_approval_rules_worker_concurrency_limit_scopeon GitLab.com through #636157, then watchsidekiq_concurrency_limit_scope_deferred_jobs_total,sidekiq_concurrency_limit_scope_unattributed_jobs_totaland this worker's queue latency.