Track all workers concurrency jobs

What does this MR do and why?

Before this, we only track the number of concurrent jobs when concurrency_limit attribute is defined on the worker class.

Now, we track all worker classes concurrent jobs in order to support the work for circuit breaking/throttling all sidekiq workers based on database usage.

This is primarily done by setting concurrency_limit to 0 by default for all workers.

This change is also behind track_sidekiq_concurrency_limit_execution feature flag.

References

gitlab-com/gl-infra/data-access/durability/team#135

What is concurrency limit in sidekiq - https://docs.gitlab.com/development/sidekiq/worker_attributes/#concurrency-limit

Screenshots or screen recordings

Before After

How to set up and validate locally

  1. Enable the feature flag track_sidekiq_concurrency_limit_execution

  2. Restart sidekiq

  3. In rails console:

    5.times { Chaos::SleepWorker.perform_async(60) }
    Gitlab::SidekiqMiddleware::ConcurrencyLimit::ConcurrencyLimitService.concurrent_worker_count(Chaos::SleepWorker.name)
  4. The above should return 5

  5. Metric should be emitted too. For some reason, I don't see the metric anymore in my GDK, but it appeared previously. Please help me to confirm this.

    curl -s 'gdk.test:3807/metrics' | rg sidekiq_concurrency_limit_current_concurrent_jobs_total

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Marco Gregorius

Merge request reports

Loading