Bound scope-set reads in the concurrency limit queue manager

What does this MR do and why?

Each resume pass of a scoped worker read the whole set of queued scopes with SMEMBERS, but a pass can serve at most limit scopes. The other members were read and shuffled for nothing, and in eager mode this repeats every pass. queue_size and drop_jobs! read the same set the same way. This MR bounds those reads, as suggested in !264227 (comment 4126254447) and tracked in #637499 (parts 2 and 3).

  • Resume pass. Draws [limit * 2, 1000].min scopes with SRANDMEMBER, skips scopes at their share, and draws again while slots remain, at most 3 draws per pass. The main queue sits at a random position among the drawn queues. A pass now costs O(free slots) instead of O(scopes queued).
  • Small backlogs are unchanged. When fewer scopes are queued than requested, SRANDMEMBER returns the whole set, so the pass reads it once and serves it as before.
  • queue_size. Exact while at most 1,000 scopes are queued. Above that, it reads 1,000 sampled scope queues and extrapolates by SCARD. It feeds the sidekiq_concurrency_limit_queue_jobs gauge and the resume stats log, which can take an estimate.
  • drop_jobs!. Has to visit every scope, so it scans the set with SSCAN in batches instead of one SMEMBERS.

Fairness is the same on average: a uniform sample is the prefix of a shuffle, so a scope still waits about (scopes queued ÷ free slots) passes.

The invariants listed in !264227 (merged) still hold. Every scoped start still goes through the two admission scripts, and scope queues are still drained when the scope is turned off, because the sampling ignores whether the worker is scoped.

Part 1 of #637499 (O(1) per-scope admission with counters) is not in this MR.

References

How to set up and validate locally

bin/rspec spec/lib/gitlab/sidekiq_middleware/concurrency_limit/queue_manager_spec.rb

The new examples cover a pass that draws the whole set once, 20 scopes served 4 per pass with every draw bounded and no SMEMBERS, the redraw bound when the drawn scopes are at their share, queue_size extrapolation, and drop_jobs! using SSCAN.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Alan (Maciej) Paruszewski

Merge request reports

Loading
Loading