Add concurrency limit scope to pipeline execution wave two workers
What does this MR do and why?
Declares concurrency_limit_scope (!264227 (merged)) on two more Pipeline Execution workers, so one top-level group is capped at a share of each worker's Sidekiq concurrency limit instead of being able to fill it:
| Worker | Scope key | Share of the worker limit |
|---|---|---|
Ci::BuildFinishedWorker |
top-level group | 30% |
Ci::CancelRedundantPipelinesWorker |
top-level group | 40% |
Ci::LowUrgencyCancelRedundantPipelinesWorker (inherits it from Ci::CancelRedundantPipelinesWorker) |
top-level group | 40% |
This is the second wave of !268561 (merged), which scoped the three lower-volume workers. Same attribute, same share rule, a new flag.
- Nothing changes until two flags are on. The scope applies only while the instance-wide
sidekiq_concurrency_limit_scopeflag (#636157, not enabled anywhere yet) and the newci_pipeline_execution_concurrency_limit_scope_wave_twoflag (rollout: #638936) are both enabled. One flag covers the three workers. Each worker keeps its owndisable_sidekiq_concurrency_limit_scope_<WorkerClass>kill switch, so they can still be enabled or stopped one at a time. - Enabling waits for O(1) scoped admission. Both workers run at a much higher rate than the wave one workers. Scoped admission today scans the worker's whole executing hash in Redis on every job, so its cost grows with the job rate times the limit. #637499 (part 1) replaces that scan with per-scope counters and must be deployed before the wave two flag is enabled in production. The rollout issue lists it as a prerequisite. The code can merge now behind the default-off flag.
- A share, not an absolute number.
max_percentage:is a share of the worker's own limit, which on GitLab.com is derived from the shard's capacity (25% of the shard for low urgency, 35% for high). 40% of the low-urgency default and 30% of the high-urgency default both come to about 10% of the shard per top-level group, the same bound wave one used. No environment variable or setting is needed, and the share follows shard resizes. - Keys follow the enqueue context.
Ci::BuildFinishedWorkeris enqueued from the build's status transition andCi::CancelRedundantPipelinesWorkerfrom the pipeline creation chain, both under the project's context, so they key on the top-level group. Jobs with no top-level group in context stay under the worker limit only and are counted insidekiq_concurrency_limit_scope_unattributed_jobs_total. Ci::LowUrgencyCancelRedundantPipelinesWorkerinherits the scope. It subclassesCi::CancelRedundantPipelinesWorkerfor scheduled pipelines, and worker attributes are inherited throughGitlab::ClassAttributes, so it is scoped too, at 40% of its own limit. It has its owndisable_sidekiq_concurrency_limit_scope_Ci::LowUrgencyCancelRedundantPipelinesWorkerkill switch.- Jobs over a share are deferred to the group's own queue and resumed by
ConcurrencyLimit::ResumeWorker. They are delayed, not dropped. A scoped job waits only behind queued jobs of its own group, or when the whole worker is at its limit, so one group's backlog does not hold back the others.
The shared example a worker with a concurrency limit scope is the one introduced by !268561 (merged), added here unchanged so this MR does not depend on it. If that MR merges first, this one picks up the same file with no conflict.
References
- https://gitlab.com/groups/gitlab-org/-/work_items/24422 (parent, confidential)
- https://gitlab.com/gitlab-org/gitlab/-/work_items/636869 (confidential)
- https://gitlab.com/gitlab-org/gitlab/-/work_items/637176 (confidential)
- Rollout issue: #638936
- Wave one: !268561 (merged) and #638713
- #637499 (O(1) per-scope admission, prerequisite for enabling)
- !264227 (merged) (adds
concurrency_limit_scope)
Screenshots or screen recordings
No UI change.
How to set up and validate locally
-
Give the workers a limit above zero and enable the flags. In a Rails console:
Feature.enable(:sidekiq_concurrency_limit_default_calculation) Feature.enable(:sidekiq_concurrency_limit_scope) Feature.enable(:ci_pipeline_execution_concurrency_limit_scope_wave_two)Then restart Rails and Sidekiq with a fleet size, for example
GITLAB_SIDEKIQ_MAX_REPLICAS=1 SIDEKIQ_CONCURRENCY=10. -
Check that the scope resolves. With the fleet above,
Ci::BuildFinishedWorker's own limit is 4 (35% of 10 threads, rounded up) andCi::CancelRedundantPipelinesWorker's is 3 (25%), so each top-level group gets 2 slots on either:scope = Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(Ci::BuildFinishedWorker) scope.type # => :root_namespace scope.limit_for(4) # => 2 Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(Ci::CancelRedundantPipelinesWorker).limit_for(3) # => 2 Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(Ci::LowUrgencyCancelRedundantPipelinesWorker).type # => :root_namespaceScope.forreturnsnilwhile any of the flags is off, or after enabling the worker'sdisable_sidekiq_concurrency_limit_scope_<WorkerClass>kill switch. -
Run a pipeline with more jobs than the share allows in one project, and watch
log/sidekiq.logforconcurrency_limitdeferrals ofCi::BuildFinishedWorkerwithconcurrency_limit_scopeset toroot_namespace:<group path>. Jobs from a project in a second top-level group start right away.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.