Track all workers concurrency jobs
What does this MR do and why?
Before this, we only track the number of concurrent jobs when concurrency_limit attribute is defined on the worker class.
Now, we track all worker classes concurrent jobs in order to support the work for circuit breaking/throttling all sidekiq workers based on database usage.
This is primarily done by setting concurrency_limit to 0 by default for all workers.
This change is also behind track_sidekiq_concurrency_limit_execution feature flag.
References
gitlab-com/gl-infra/data-access/durability/team#135
What is concurrency limit in sidekiq - https://docs.gitlab.com/development/sidekiq/worker_attributes/#concurrency-limit
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
-
Enable the feature flag
track_sidekiq_concurrency_limit_execution -
Restart sidekiq
-
In rails console:
5.times { Chaos::SleepWorker.perform_async(60) } Gitlab::SidekiqMiddleware::ConcurrencyLimit::ConcurrencyLimitService.concurrent_worker_count(Chaos::SleepWorker.name) -
The above should return
5 -
Metric should be emitted too. For some reason, I don't see the metric anymore in my GDK, but it appeared previously. Please help me to confirm this.
curl -s 'gdk.test:3807/metrics' | rg sidekiq_concurrency_limit_current_concurrent_jobs_total
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.