Bound scope-set reads in the concurrency limit queue manager
What does this MR do and why?
Each resume pass of a scoped worker read the whole set of queued scopes with SMEMBERS, but a pass can serve at most limit scopes. The other members were read and shuffled for nothing, and in eager mode this repeats every pass. queue_size and drop_jobs! read the same set the same way. This MR bounds those reads, as suggested in !264227 (comment 4126254447) and tracked in #637499 (parts 2 and 3).
- Resume pass. Draws
[limit * 2, 1000].minscopes withSRANDMEMBER, skips scopes at their share, and draws again while slots remain, at most 3 draws per pass. The main queue sits at a random position among the drawn queues. A pass now costs O(free slots) instead of O(scopes queued). - Small backlogs are unchanged. When fewer scopes are queued than requested,
SRANDMEMBERreturns the whole set, so the pass reads it once and serves it as before. queue_size. Exact while at most 1,000 scopes are queued. Above that, it reads 1,000 sampled scope queues and extrapolates bySCARD. It feeds thesidekiq_concurrency_limit_queue_jobsgauge and the resume stats log, which can take an estimate.drop_jobs!. Has to visit every scope, so it scans the set withSSCANin batches instead of oneSMEMBERS.
Fairness is the same on average: a uniform sample is the prefix of a shuffle, so a scope still waits about (scopes queued ÷ free slots) passes.
The invariants listed in !264227 (merged) still hold. Every scoped start still goes through the two admission scripts, and scope queues are still drained when the scope is turned off, because the sampling ignores whether the worker is scoped.
Part 1 of #637499 (O(1) per-scope admission with counters) is not in this MR.
References
- #637499
- !264227 (merged) (parent MR; review thread !264227 (comment 4126254447))
How to set up and validate locally
bin/rspec spec/lib/gitlab/sidekiq_middleware/concurrency_limit/queue_manager_spec.rbThe new examples cover a pass that draws the whole set once, 20 scopes served 4 per pass with every draw bounded and no SMEMBERS, the redraw bound when the drawn scopes are at their share, queue_size extrapolation, and drop_jobs! using SSCAN.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.