Count concurrency limit queues exactly in queue_size

What does this MR do and why?

Follow-up from !267044 (comment 4170219572).

Past 1,000 queued scopes, QueueManager#queue_size extrapolated the scope queues from a random sample. With uneven scope queues that estimate swings far from the real total. That makes the sidekiq_concurrency_limit_queue_jobs gauge, and the alerts that read it through max_over_time, unreliable.

  • queue_size is now an exact count. It reuses each_queue, the batched SSCAN that drop_jobs! already uses, and pipelines the LLENs 1,000 at a time. The sampler, the admin API, the Duo credits cron log and the search rake task all get the exact count.
  • The sampling code moves unchanged into estimated_queue_size. Only ConcurrencyLimit::ResumeWorker uses it, because it can run about once a second per worker during a backlog. Its stats log key changes from concurrency_limit_queue_size to concurrency_limit_queue_size_estimate, so nobody reads the value as exact.
  • Resume decisions still use the exact has_jobs_in_queue?, so throughput doesn't change.

No feature flag and no changelog entry: unscoped workers have an empty scopes set and count the same as before, and scope queues only fill behind the default-off sidekiq_concurrency_limit_scope flag.

References

Screenshots or screen recordings

Recorded in local GDK. A test worker has a real 60,000-job backlog: 5 namespaces with 10,000 jobs each, plus 5,000 namespaces with 2 each. Both methods read the same Redis data on every tick.

Before (estimated_queue_size, 20 ticks) After (queue_size, 20 ticks)
10,010 to 110,090; 6 of 20 ticks over the 100k alert 60,000 every tick

Cost at 100,000 queued scopes, averaged over 5 calls: about 59 ms of Redis server time per call (36 ms LLEN, 23 ms SSCAN, from an INFO commandstats delta), and about 340 ms of wall time in the calling process. The sampler runs every 30 s.

How to set up and validate locally

  1. In bin/rails console, fill a fake worker's queues directly in Redis (no worker class needed):

    svc = Gitlab::SidekiqMiddleware::ConcurrencyLimit::ConcurrencyLimitService
    qm = Gitlab::SidekiqMiddleware::ConcurrencyLimit::QueueManager.new(worker_name: 'DemoWorker', prefix: svc::REDIS_KEY_PREFIX)
    scopes = qm.redis_key.sub(':throttled_jobs:', ':throttled_scopes:')
    jobs = (1..5).to_h { |i| ["heavy#{i}", 10_000] }.merge((1..5_000).to_h { |i| ["ns#{i}", 2] })
    jobs.each_slice(500) do |batch|
      Gitlab::Redis::ConcurrencyLimit.with do |r|
        r.pipelined { |p| batch.each { |k, n| p.rpush("#{qm.redis_key}:scope:#{k}", Array.new(n, '{}')); p.sadd(scopes, k) } }
      end
    end
  2. Compare the two counts a few times:

    5.times { p [svc.queue_size('DemoWorker'), svc.estimated_queue_size('DemoWorker')] }
    # => [60000, 10010], [60000, 110090], [60000, 60050], ...
  3. Clean up:

    Gitlab::Redis::ConcurrencyLimit.with { |r| r.del(*jobs.keys.map { |k| "#{qm.redis_key}:scope:#{k}" }, scopes) }

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Reviewable LOC

About 23 lines of application code changed (app/, lib/) and 42 lines of specs.

Context for LLM agents

  • queue_size can count a scope twice if SSCAN returns it twice while the set rehashes. That's accepted: deduping would cost memory per scope just to keep a gauge exact.
  • MAX_SCOPE_SAMPLE is both the SSCAN batch size and the estimate's sample size, as before.

Owed elsewhere

  • Any Kibana view on the old concurrency_limit_queue_size log key needs to switch to concurrency_limit_queue_size_estimate. Nothing in this repo references the old key. I haven't checked runbooks or dashboards.

Merge request reports

Loading
Loading