Run new jobs while a worker is below its concurrency limit

What does this MR do and why?

ConcurrencyLimit::Middleware defers every new job while the worker's deferred list is non-empty, however few jobs are running. A brief spike to the limit therefore latches a backlog that drains only at the ConcurrencyLimit::ResumeWorker rate. On 2026-10-08 Ci::BuildTraceChunkFlushWorker ran 988 of 2,475 slots while deferring 260–680 jobs/s for 100 minutes, and Ci::ExecuteBuildHooksWorker ran 402 of 1,500 while its backlog grew to 1.65M (INC-14968).

This MR adds a per-worker opt-in, the worker type flag sidekiq_concurrency_limit_use_free_slots_<WorkerName>. While it is enabled for a worker:

  • A new job runs whenever the worker is below its limit, even while deferred jobs wait. It is deferred only when the worker is at its limit. The check and the tracking happen in one Lua step (TRACK_SCOPED_EXECUTION_START_SCRIPT from !264227 (merged), called with no scope), so jobs picked up together cannot all pass.
  • Resumed jobs reserve their slot when ConcurrencyLimit::ResumeWorker picks them, through the same reservation path as scoped workers. Reservations count against the limit in the executing hash, so a new job cannot take a slot the resumer has already claimed for an older deferred job.
  • Once the oldest deferred job has waited 5 minutes, new jobs leave 10% of the limit (at least one slot) free for resumed jobs. This is a throughput floor for the deferred list under sustained overload, not a re-latch: while the worker has free slots beyond the reserve, new jobs still run, so a long backlog on a half-idle worker drains at the resumer's rate instead of growing. Resumed jobs reserve against the full limit.
  • ConcurrencyLimit::ResumeWorker keeps rescheduling itself every second while jobs are deferred, instead of returning when the worker is at its limit. In free-slot mode a full worker is the normal state under load, and new jobs fill freed slots between runs, so waiting for the cron would slow the drain to one batch a minute. ResumeWorker is idempotent!, so duplicate-job deduplication coalesces a self-scheduled poll with a cron run for the same worker, and return unless has_jobs_in_queue? ends the chain once the list drains.

The limit stays a hard bound. Flag off, or an empty deferred list, behaves exactly as today. While the worker's concurrency_limit_scope is on, the flag has no effect, because its jobs already wait only behind their own scope. With the scope turned off, the flag applies as it does to an unscoped worker, including to jobs still waiting in its scope queues. The flag requires sidekiq_concurrency_limit_middleware to be on, like the scope, and is checked against :instance so enqueue, perform and resume agree.

Also adds "Events might arrive out of order" to the Job events webhook docs, as agreed with the Pipeline Authoring group for Ci::ExecuteBuildHooksWorker.

Why opt-in per worker

The non-empty-list deferral is a deliberate ordering guarantee from !139851 (merged), and worker_attributes.md documents it. Some workers may rely on it. Pipeline Execution confirmed Ci::BuildTraceChunkFlushWorker and Ci::SyncReportsToReportApprovalRulesWorker are order independent, and Pipeline Authoring accepted reordering for Ci::ExecuteBuildHooksWorker while a backlog exists. Those are the first candidates; enabling is a runtime ChatOps action, not a code change.

Rejected alternatives

  • Count executing plus the pending_resumed counter instead of reservations. The earlier prototype did this. It leaves the check-then-act race between new arrivals and resumed jobs, and the counter drifts. The reservation path from !264227 (merged) is exact and already handles the hand-off, cleanup and release cases.
  • Bring back the latch when the head job is older than the cap. A long backlog on a half-idle worker also has an old head. Re-latching there would send new jobs to the list while the worker has free slots, and if the resumer is slower than arrivals (430–660/s vs 520–620/s on 2026-10-08) the list grows and the latch never releases. That is the incident this MR fixes. A reserve for the resumer bounds starvation without that failure mode.
  • Swap a new job for the head of the list. Preserves FIFO but roughly triples the Redis operations per job and makes the middleware a second writer to the deferred list.

Dependency

Built on !264227 (merged), merged to master on 2026-11-03, because it reuses the atomic admission script and the reservation model. The branch is rebased onto master, so the diff is this MR's 5 commits only.

That MR changes the limiter with the flag off too. Per !266230 (comment 4154010530), wait until it reaches workflowproduction and has baked for one working day before merging this one.

!267110 (merged) changes TRACK_SCOPED_EXECUTION_START_SCRIPT so a resumed job takes over only a reservation of its own scope. This MR calls the script with no scope (ARGV[4] == ''), so the free-slot takeover is unchanged.

Known limitation: reservation expiry

A reservation is stale after TRACKING_KEY_TTL (600s), as raised in !264227 (comment 4140110342). If a resumed job waits in Sidekiq longer than that, ResumeWorker can hand its slot to the next batch. In free-slot mode the late job then runs if the worker is below its limit, or is deferred to the tail of the list. The limit still holds. A separate reservation TTL and a counter for this case are tracked in #637499.

A resumed job lost outside the middleware chain (the Sidekiq queue cleared during an incident, data loss on the Sidekiq Redis shard, a queue nobody listens to) leaves its resumed:<jid> reservation in the executing hash until cleanup_stale_trackers removes it, 10–20 minutes later. Every admission check counts it, so with the flag on each lost job holds back one slot from new jobs as well as from resuming, even while the deferred list is empty. Today new jobs are unaffected. Turning the free-slot flag off does not recover the slot, because the flag-off check counts HLEN too. Recovery: enable disable_sidekiq_concurrency_limit_middleware_<WorkerName>, or delete the resumed:* fields from sidekiq:concurrency_limit:{<worker>}:executing. The symptom is sidekiq_concurrency_limit_current_concurrent_jobs at the limit while far fewer jobs run. Scoped workers have had this since !264227 (merged); see !266230 (comment 4171007800). The reservation TTL from #637499 lands before Ci::BuildTraceChunkFlushWorker is enabled.

References

Screenshots or screen recordings

Not applicable: Sidekiq middleware change.

How to set up and validate locally

  1. Add a throwaway worker, for example app/workers/free_slot_demo_worker.rb:

    class FreeSlotDemoWorker
      include ApplicationWorker
    
      feature_category :scalability
      data_consistency :sticky
      urgency :low
    
      concurrency_limit -> { 5 }
    
      def perform(_i)
        sleep 10
      end
    end
  2. Restart Sidekiq: gdk restart rails-background-jobs.

  3. Fill the limit and start a backlog, then let the running jobs finish:

    8.times { |i| FreeSlotDemoWorker.perform_async(i) }
    sleep 12
    svc = Gitlab::SidekiqMiddleware::ConcurrencyLimit::ConcurrencyLimitService
    svc.concurrent_worker_count('FreeSlotDemoWorker') # => 0
    svc.queue_size('FreeSlotDemoWorker')              # => 3
  4. With the flag off, a new job is deferred behind the three queued ones despite the idle worker:

    FreeSlotDemoWorker.perform_async(99)
    svc.queue_size('FreeSlotDemoWorker') # => 4
  5. Enable the flag and enqueue again. The job runs while the three remain queued:

    Feature.enable(:sidekiq_concurrency_limit_use_free_slots_FreeSlotDemoWorker)
    FreeSlotDemoWorker.perform_async(100)
    svc.concurrent_worker_count('FreeSlotDemoWorker') # => 1
    svc.queue_size('FreeSlotDemoWorker')              # => 4
  6. Resume the queued jobs by hand (ResumeWorker only schedules workers listed in all_queues.yml) and confirm they reserve their slots up to the limit:

    ConcurrencyLimit::ResumeWorker.new.perform('FreeSlotDemoWorker')
    svc.concurrent_worker_count('FreeSlotDemoWorker') # => 5, including reservations

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

  • Specs cover the client, server, service, queue manager and resume worker paths, including the reservation hand-off, the pending-counter fallback when the flag is toggled with jobs in flight, and the age-cap reserve. spec/lib/gitlab/sidekiq_middleware/concurrency_limit/ and spec/workers/concurrency_limit/ pass locally: 330 examples, 0 failures. Each behaviour is tested at one layer: the reserve arithmetic in free_slots_spec.rb, the Redis wiring in concurrency_limit_service_spec.rb, and the deferral decision in the middleware specs.
  • Documentation updated in doc/development/sidekiq/worker_attributes.md and doc/user/project/integrations/webhook_events.md.
  • Feature flag: worker type, no YAML definition, disabled by default, same pattern as disable_sidekiq_concurrency_limit_middleware_<WorkerName>.
Edited by Efeosa Eguavoen

Merge request reports

Loading
Loading