Set up a way to override concurrency limit

Concurrency Limit currently is defined by:

• Hardcoded in worker attributes, eg https://gitlab.com/gitlab-org/gitlab/blob/main/ee/app/workers/security/process_scan_result_policy_worker.rb#L14-14
• Dynamically adjusted based on DB usage, gitlab-com/gl-infra/observability/team#3815 (closed) (example implementation gitlab-org/gitlab!190565 (diffs))

Once the concurrency limit is dynamic, we need a way to override the current limit, for cases like:

• A worker is over-throttled causing large queue backlog even though DB is fine -> Recover the current limit back to its max limit
• DB having an issue -> all workers being throttled (because DB duration spiked) -> DB back to healthy -> instantly recover the current limit back to its max limit
  • We probably need a special admin API that accepts query selector (like in routing rules) to update many workers at once easily

Ideas

1. Extend the Sidekiq Web UI /admin/sidekiq to list down the workers and current limit, with field to override the current limit in Redis.

   • Design example: gitlab-org/gitlab!190565 (comment 2554332920)

     

2. ApplicationSettings
   • Not ideal because Redis is the SoT for the current limit
3. ChatOps via dedicated admin API. We'll focus on building this first.