Count concurrency limit queues exactly in queue_size
What does this MR do and why?
Follow-up from !267044 (comment 4170219572).
Past 1,000 queued scopes, QueueManager#queue_size extrapolated the scope queues from a random sample. With uneven scope queues that estimate swings far from the real total. That makes the sidekiq_concurrency_limit_queue_jobs gauge, and the alerts that read it through max_over_time, unreliable.
queue_sizeis now an exact count. It reuseseach_queue, the batchedSSCANthatdrop_jobs!already uses, and pipelines theLLENs 1,000 at a time. The sampler, the admin API, the Duo credits cron log and the search rake task all get the exact count.- The sampling code moves unchanged into
estimated_queue_size. OnlyConcurrencyLimit::ResumeWorkeruses it, because it can run about once a second per worker during a backlog. Its stats log key changes fromconcurrency_limit_queue_sizetoconcurrency_limit_queue_size_estimate, so nobody reads the value as exact. - Resume decisions still use the exact
has_jobs_in_queue?, so throughput doesn't change.
No feature flag and no changelog entry: unscoped workers have an empty scopes set and count the same as before, and scope queues only fill behind the default-off sidekiq_concurrency_limit_scope flag.
References
- Issue and approved plan: #638663 (comment 4190544248)
- Scope rollout: #636157
Screenshots or screen recordings
Recorded in local GDK. A test worker has a real 60,000-job backlog: 5 namespaces with 10,000 jobs each, plus 5,000 namespaces with 2 each. Both methods read the same Redis data on every tick.
Before (estimated_queue_size, 20 ticks) |
After (queue_size, 20 ticks) |
|---|---|
| 10,010 to 110,090; 6 of 20 ticks over the 100k alert | 60,000 every tick |
Cost at 100,000 queued scopes, averaged over 5 calls: about 59 ms of Redis server time per call (36 ms LLEN, 23 ms SSCAN, from an INFO commandstats delta), and about 340 ms of wall time in the calling process. The sampler runs every 30 s.
How to set up and validate locally
-
In
bin/rails console, fill a fake worker's queues directly in Redis (no worker class needed):svc = Gitlab::SidekiqMiddleware::ConcurrencyLimit::ConcurrencyLimitService qm = Gitlab::SidekiqMiddleware::ConcurrencyLimit::QueueManager.new(worker_name: 'DemoWorker', prefix: svc::REDIS_KEY_PREFIX) scopes = qm.redis_key.sub(':throttled_jobs:', ':throttled_scopes:') jobs = (1..5).to_h { |i| ["heavy#{i}", 10_000] }.merge((1..5_000).to_h { |i| ["ns#{i}", 2] }) jobs.each_slice(500) do |batch| Gitlab::Redis::ConcurrencyLimit.with do |r| r.pipelined { |p| batch.each { |k, n| p.rpush("#{qm.redis_key}:scope:#{k}", Array.new(n, '{}')); p.sadd(scopes, k) } } end end -
Compare the two counts a few times:
5.times { p [svc.queue_size('DemoWorker'), svc.estimated_queue_size('DemoWorker')] } # => [60000, 10010], [60000, 110090], [60000, 60050], ... -
Clean up:
Gitlab::Redis::ConcurrencyLimit.with { |r| r.del(*jobs.keys.map { |k| "#{qm.redis_key}:scope:#{k}" }, scopes) }
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Reviewable LOC
About 23 lines of application code changed (app/, lib/) and 42 lines of specs.
Context for LLM agents
queue_sizecan count a scope twice ifSSCANreturns it twice while the set rehashes. That's accepted: deduping would cost memory per scope just to keep a gauge exact.MAX_SCOPE_SAMPLEis both theSSCANbatch size and the estimate's sample size, as before.
Owed elsewhere
- Any Kibana view on the old
concurrency_limit_queue_sizelog key needs to switch toconcurrency_limit_queue_size_estimate. Nothing in this repo references the old key. I haven't checked runbooks or dashboards.