Add per-namespace concurrency limit scope for Sidekiq workers

What does this MR do and why?

A Sidekiq worker's concurrency limit is shared by every tenant. One top-level group that enqueues a burst of jobs fills the whole budget and defers every other group's jobs behind its backlog. This happened with Security::ScanResultPolicies::SyncMergeRequestApprovalsWorker, as described in the related issue.

This MR adds an opt-in concurrency_limit_scope worker attribute to the concurrency limit middleware. It caps each top-level group or user at a share of the worker's limit, at an absolute limit, or at the lower of both:

concurrency_limit_scope(:root_namespace, max_percentage: 0.3,
  enabled: -> { Feature.enabled?(:my_worker_scoped_concurrency_limit, :instance) })
concurrency_limit_scope(:user, limit: -> { Gitlab::CurrentSettings.some_per_user_limit })
  • The worker-wide limit stays a hard bound across all scopes. New jobs are admitted, and resumed jobs reserve their slot, in one Lua step against the executing hash. A resumed job whose reservation was cleaned up while it waited in Sidekiq is checked again like a new job.
  • One scope's backlog does not hold back the others. Deferred jobs of a scoped worker wait in a queue per scope. ConcurrencyLimit::ResumeWorker serves the queues in random order, each up to its free share. A job waits on enqueue only behind jobs of its own scope, or behind any queued jobs while the worker is at its limit.
  • No lookup on enqueue. A job counts against the top-level group or user in the application context it was enqueued with (meta.root_namespace, meta.gl_user_id), which Labkit writes on the job anyway. A scope with only a limit: that returns 0 or less caps nothing, so the worker stays unscoped.
  • Three switches. A worker's scope applies only while the instance-wide sidekiq_concurrency_limit_scope beta flag is on, the worker-type kill switch disable_sidekiq_concurrency_limit_scope_<WorkerName> is off, and the optional enabled: lambda returns true. The kill switch follows disable_sidekiq_concurrency_limit_middleware_<WorkerName> and turns off only the scope, leaving the worker limit on. enabled: lets an adopter roll out on its own flag. No worker declares the attribute yet, so this MR changes nothing.
  • Coverage is measured. sidekiq_concurrency_limit_scope_unattributed_jobs_total counts the jobs of a scoped worker that cannot be attributed, for example when the context has no root namespace. Those jobs fall under the worker limit only.

Reviewable LOC: 614 (plus 995 in tests)

The pieces depend on each other to keep the limit a hard bound and to avoid starving small scopes: admission, reservations, per-scope queues and the enqueue check. A split would leave the first MR's guarantee broken until the second merged. Review the full diff rather than commit by commit: later commits add and then remove a log-only mode and resuming when a job ends, and remove and then restore the instance flag.

References

Screenshots or screen recordings

A recorded before/after run in GDK is attached as an internal note. It uses real Sidekiq, a worker limited to 10, and a top-level group with 100,001 projects enqueuing one job per project, while two other groups each send one job per second. Every job is enqueued under its group's application context, so no project is looked up.

Before (scope off) After (scope on, 30% share)
The big group takes all 10 slots, and the worker reaches 13, the existing middleware's check-then-act race. Jobs of the other groups stop running: 5 start in the 134 seconds after the burst, one after waiting 73s. The big group stays at 3 slots, and the worker at 9 or fewer. The other groups' jobs wait 0.02s (median of 376).

During the first ~35 seconds, the other groups waited up to 15s. That's the first bulk batches, already in Sidekiq before the big group had a backlog. On GDK it takes that long to defer them. While the rest of the burst was still being enqueued, which took 121s on this run, a few of their jobs waited up to 3s.

How to set up and validate locally

  1. Add a throwaway worker, for example app/workers/scoped_demo_worker.rb:

    class ScopedDemoWorker
      include ApplicationWorker
    
      feature_category :security_policy_management
      data_consistency :sticky
      urgency :low
    
      concurrency_limit -> { 5 }
      concurrency_limit_scope(:root_namespace, max_percentage: 0.4)
    
      def perform(_project_id)
        sleep 3
      end
    end
  2. Enable the flag: Feature.enable(:sidekiq_concurrency_limit_scope).

  3. Restart Sidekiq: gdk restart rails-background-jobs.

  4. Enqueue a burst for one top-level group, then a few jobs for another:

    a, b = Group.where(parent_id: nil).select { |g| g.all_projects.exists? }.first(2).map { |g| g.all_projects.first }
    Gitlab::ApplicationContext.with_context(project: a) { 12.times { ScopedDemoWorker.perform_async(a.id) } }
    Gitlab::ApplicationContext.with_context(project: b) { 3.times { ScopedDemoWorker.perform_async(b.id) } }
  5. Watch the executing entries per scope. The first group stays at 2 and the total stays at or under 5, while the second group's jobs start right away:

    svc = Gitlab::SidekiqMiddleware::ConcurrencyLimit::ConcurrencyLimitService
    10.times { p svc.concurrent_counts_by_scope('ScopedDemoWorker'); ConcurrencyLimit::ResumeWorker.perform_async('ScopedDemoWorker'); sleep 1 }

    ConcurrencyLimit::ResumeWorker only schedules workers listed in all_queues.yml, so the loop resumes the throwaway worker by hand.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Context for LLM agents

Rejected alternatives:

  • Rotating skipped jobs through one shared deferred queue. The first version did this. In the recorded flood, a small group's deferred job sat behind 99k jobs, past the 5,000-entry resume window, for 254s. Per-scope queues replaced it.
  • Admission on the server only. All 100k jobs of the burst went through the Sidekiq queue once before being deferred. Other groups' jobs, and any other worker on that queue, waited up to 250s behind them.
  • A short-lived "backlog" marker key, and per-scope pending counters with TTLs. Both drifted. A refreshed TTL never decays a leaked count, and a fixed TTL expires live counts. The exact signals are the scopes set (SISMEMBER) and per-job reservations in the executing hash, which the existing stale-tracker cleanup removes one by one.
  • A separate per-actor middleware deferring to the schedule set, as in !264233 (closed). Its backoff (30s to 5 min) misses the 90-second unblock deadline. It stops counting a job queued for over 15 minutes. It also leaves the worker-wide buffer shared, which is where the incident's backlog built up. This MR takes two of its ideas instead: the tenant from the job context and an absolute limit.
  • A per-actor middleware with its own deferred list per tenant, as in !264314 (closed). It runs after the worker-wide limiter, so a burst is still parked in the shared worker queue before the tenant check sees it. A pushed job stops counting after 15 minutes, and its limit is absolute only, with no share of the worker limit. Its resume when a job ends was tried here and removed: it reserved freed slots for jobs still behind a burst in Sidekiq, and fixing that needed a takeover rule for reservations and a resume from inside the finished job, for a gain only to the scope over its share.
  • Only a per-worker enabled: lambda, with no instance flag. It left no switch to turn every scoped worker off at once, and no way to stop one worker's scope without a deploy when it declares no lambda. The instance flag and the per-worker kill switch cover both. enabled: stays optional, for an adopter that wants its own rollout.
  • A log-only mode behind a second flag. It counted only the share check when a job started, not the deferral behind a scope's own queue on enqueue, so it reported fewer deferrals than enforcement makes. The Sidekiq job logs already carry the root namespace and user, so per-tenant concurrency can be measured there. A first rollout can use a generous max_percentage: instead.
  • Adapting the cap to the number of active scopes. A tenant bursting alone would hold every slot, and the next tenant would wait for those jobs to finish. That's too slow against the 90-second unblock deadline.

Invariants a follow-up must keep:

  • Every path that starts a scoped job goes through TRACK_SCOPED_EXECUTION_START_SCRIPT or RESERVE_RESUMED_JOBS_SCRIPT. Each one checks the worker limit against HLEN of the executing hash, which includes reservations (resumed:<jid> fields).
  • Scope.for returns nil unless sidekiq_concurrency_limit_scope is on, disable_sidekiq_concurrency_limit_scope_<WorkerName> is off, the worker's enabled: (when given) returns true, and sidekiq_concurrency_limit_middleware is on. The scope flags and enabled: must not depend on the job (checked against :instance only), because enqueue, perform and resume must agree on whether a worker is scoped.
  • A key is the enqueuing code's tenant, which can differ from the job's subject (for example, a fork). Keys are root_namespace:<full path> and user:<id>, so renaming a top-level group splits its share across two keys while its old jobs drain.
  • Scope queues are drained even when the scope is turned off, so no deferred job is stranded.

Known behavior changes for scoped workers only:

  • sidekiq_concurrency_limit_current_concurrent_jobs and the resume worker_stats_log count reserved, not-yet-started jobs.
  • A scope's deferred jobs are resumed only by ConcurrencyLimit::ResumeWorker, as for unscoped workers: the cron starts it within a minute, and it reruns every second while jobs wait and the worker has free slots.
  • Per-namespace deferral shows in the deferred job log as concurrency_limit_scope. The new counters sidekiq_concurrency_limit_scope_deferred_jobs_total and sidekiq_concurrency_limit_scope_unattributed_jobs_total are labelled by worker and scope type only, to keep metric cardinality bounded.

Owed elsewhere

  • Opt in the security policy workers listed in #633462. That issue stays open for this. The follow-up must enqueue them under the tenant's application context, since scopes read only the job context.
  • An adopting worker that wants its own rollout brings its own enabled: flag and rollout issue.
Edited by Alan (Maciej) Paruszewski

Merge request reports

Loading
Loading