Add concurrency limit scope to pipeline execution wave one workers
What does this MR do and why?
Declares concurrency_limit_scope (!264227 (merged)) on three Pipeline Execution workers, so one top-level group or user is capped at a share of each worker's Sidekiq concurrency limit instead of being able to fill it:
| Worker | Scope key | Share of the worker limit |
|---|---|---|
Ci::CreateDownstreamPipelineWorker |
top-level group | 40% |
MergeTrains::RefreshWorker |
top-level group | 40% |
Ci::DropPipelineWorker |
user | 30% |
Ci::DropPipelineForBlockedUserWorker (inherits it from Ci::DropPipelineWorker) |
user | 30% |
- Nothing changes until two flags are on. The scope applies only while the instance-wide
sidekiq_concurrency_limit_scopeflag (#636157, not enabled anywhere yet) and the newci_pipeline_execution_concurrency_limit_scope_wave_oneflag (rollout: #638713) are both enabled. One flag covers the three workers. Each worker keeps its owndisable_sidekiq_concurrency_limit_scope_<WorkerClass>kill switch, so they can still be enabled or stopped one at a time. - A share, not an absolute number.
max_percentage:is a share of the worker's own limit, which on GitLab.com is derived from the shard's capacity (25% of the shard for low urgency, 35% for high). 40% of the low-urgency default and 30% of the high-urgency default both come to about 10% of the shard per tenant, the bound Hordur Freyr Yngvason proposed in https://gitlab.com/gitlab-org/gitlab/-/work_items/604640 ("up to 10% of total capacity"). No environment variable or setting is needed, and the share follows shard resizes. - Keys follow the enqueue context. Downstream pipeline creation and merge train refreshes are enqueued under the project's context, so they key on the top-level group. Pipeline drops are enqueued with
bulk_perform_async_with_contexts(..., context_proc: -> { { user: context_user } }), where the context user is the banned or blocked user, so they key on that user, not on whoever triggered the drop. Jobs with no tenant in context (for exampleMergeTrains::RefreshWorkerjobs from the unstick cron) stay under the worker limit only and are counted insidekiq_concurrency_limit_scope_unattributed_jobs_total. Ci::DropPipelineForBlockedUserWorkerinherits the scope. It subclassesCi::DropPipelineWorker, and worker attributes are inherited throughGitlab::ClassAttributes, so the user-block path is scoped too: 30% of its low-urgency limit, about 7.5% of the shard, per blocked user. It has its owndisable_sidekiq_concurrency_limit_scope_Ci::DropPipelineForBlockedUserWorkerkill switch.- Jobs over a share are deferred to the tenant's own queue and resumed by
ConcurrencyLimit::ResumeWorker. They are delayed, not dropped.
These three workers run at low volume on GitLab.com. #637499 asks that only low-volume workers adopt the scope, one at a time, until scoped admission is O(1). The rollout issue enables them one at a time through the per-worker kill switches.
References
- https://gitlab.com/groups/gitlab-org/-/work_items/24422 (parent, confidential)
- https://gitlab.com/gitlab-org/gitlab/-/work_items/636881 (confidential)
- https://gitlab.com/gitlab-org/gitlab/-/work_items/637175 (confidential)
- https://gitlab.com/gitlab-org/gitlab/-/work_items/637177 (confidential)
- Rollout issue: #638713
- !264227 (merged) (adds
concurrency_limit_scope) - !264410 (merged) (same attribute on pipeline creation, per user)
Screenshots or screen recordings
No UI change.
How to set up and validate locally
-
Give the workers a limit above zero and enable the flags. In a Rails console:
Feature.enable(:sidekiq_concurrency_limit_default_calculation) Feature.enable(:sidekiq_concurrency_limit_scope) Feature.enable(:ci_pipeline_execution_concurrency_limit_scope_wave_one)Then restart Rails and Sidekiq with a fleet size, for example
GITLAB_SIDEKIQ_MAX_REPLICAS=1 SIDEKIQ_CONCURRENCY=10. -
Check that the scope resolves. With the fleet above, the worker's own limit is 3 (25% of 10 threads, rounded up), so each top-level group gets 2 slots:
scope = Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(Ci::CreateDownstreamPipelineWorker) scope.type # => :root_namespace scope.limit_for(3) # => 2 Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(Ci::DropPipelineWorker).type # => :userScope.forreturnsnilwhile any of the flags is off, or after enabling the worker'sdisable_sidekiq_concurrency_limit_scope_<WorkerClass>kill switch. -
Trigger more downstream pipelines from one project than the share allows, and watch
log/sidekiq.logforconcurrency_limitdeferrals withconcurrency_limit_scopeset toroot_namespace:<group path>. Jobs from a second top-level group start right away.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.