Emit concurrency limit stats in background sampler

Sylvester Chinrequested to merge
sc1-concurrency-limit-obs into master

What does this MR do and why?

This MR introduces a ConcurrencyLimitSampler which runs in the background to increment counters periodically. This sampler runs on all sidekiq processes. This solves 2 issues:

  • stale gauge values due to a single worker class is resolved by using a sampler so we get samples from all proceses
  • gauge values like in GlobalSearchSampler are difficult to aggregate as we would have to use a quantile approach which gives us a "confidence interval" of sorts rather than an individual value.

We can use max(idelta(xxx[1m)) to get the point-in-time queue size or concurrent count.

Also, ConcurrencyLimit::ResumeWorker now logs concurrency stats per worker.

Issue: gitlab-com/gl-infra/scalability#3823 (comment 2158893241)

Introduced in place of: !168747 (closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Edited by Sylvester Chin

Merge request reports