Add concurrency limit scope to pipeline execution wave three workers
What does this MR do and why?
Third wave of the per-tenant concurrency limit scope for Pipeline Execution workers, covering the four highest-volume ones. Three changes:
| Worker | GitLab.com shard | Worker limit (share of the shard) | Scope key | Share of the worker limit |
|---|---|---|---|---|
PipelineProcessWorker |
urgent-ci-pipeline |
60% (unchanged) | top-level group | 15% |
Ci::BuildTraceChunkFlushWorker |
urgent-other |
45% (unchanged) | top-level group | 20% |
Ci::ArchiveTraceWorker |
low-urgency-ci-pipeline |
32% -> 60% | top-level group | 15% |
Ci::ExecuteBuildHooksWorker |
catchall-b -> low-urgency-ci-pipeline |
25% default -> 45% | top-level group | 20% |
- Declares
concurrency_limit_scope(!264227 (merged)) on the four workers, so one top-level group is capped at a share of each worker's Sidekiq concurrency limit instead of being able to fill it. Same attribute and share rule as !268561 (merged) and !268941 (merged), behind a new flag. - Raises two worker limits.
Ci::ArchiveTraceWorkergoes from 32% to 60% of the shard andCi::ExecuteBuildHooksWorkergets 45% instead of the 25% low-urgency default. Both limits sit below the workers' GitLab.com peaks, which is why both workers run with the limiter bypassed there today. 60% and 45% are the values !226514 (merged) and !228036 (merged) picked forPipelineProcessWorkerandCi::BuildTraceChunkFlushWorkerfor the same reason, and both leave the same headroom over the past week's peak as those two workers have today. The other two limits already clear their peaks and stay as they are. - Moves
Ci::ExecuteBuildHooksWorkerto thecontinuous_integrationfeature category. This is the commit from !268358 (closed), cherry-picked with its author kept, because Pipeline Execution is taking the worker over and the limit above has to be sized for the shard it will run on. On GitLab.com the routing rules sendcontinuous_integration+ low urgency tolow-urgency-ci-pipeline, so this moves the worker offcatchall-b.
What each change needs before it does anything on GitLab.com:
- The scope needs two flags. The instance-wide
sidekiq_concurrency_limit_scopeflag (#636157) and the newci_pipeline_execution_concurrency_limit_scope_wave_threeflag (rollout: #639186). One flag covers the four workers; each keeps its owndisable_sidekiq_concurrency_limit_scope_<WorkerClass>kill switch so they can be enabled or stopped one at a time. Enabling in production waits for #637499 part 1 (O(1) per-scope admission), as for wave two. - The limits need the bypass flags deleted. All four workers have
disable_sidekiq_concurrency_limit_middleware_<WorkerClass>enabled on GitLab.com, which makes the middleware skip them entirely, so the new percentages are a no-op for enforcement until the rollout issue deletes those flags. Thesidekiq_concurrency_limit_max_concurrent_jobsandsidekiq_concurrency_limit_current_limitgauges for the two workers do move on the deploy, becauseConcurrencyLimitSamplerreads the configured limit without checking the bypass. That happens per worker, after !266230 (merged) (free-slot admission) is deployed and enabled for the worker, since the non-empty-list latch is what put these workers behind the bypass in the first place. - The shard move happens on deploy. It is a routing consequence of the category change, not a flag. The rollout issue lists
low-urgency-ci-pipelinesaturation as the first thing to watch.
Other notes:
- A share, not an absolute number.
max_percentage:is a share of the worker's own limit, which on GitLab.com is derived from the shard's capacity. 15% of a 60% limit and 20% of a 45% limit both come to 9% of the shard per top-level group, about the 10% bound waves one and two used. The share follows shard resizes. - Keys follow the enqueue context. All four workers are enqueued under the project's context (pipeline and job status transitions, trace chunk writes, the job-finished hand-off), so they key on the top-level group. Jobs with no top-level group in context, such as the cron-driven ones, stay under the worker limit only and are counted in
sidekiq_concurrency_limit_scope_unattributed_jobs_total. - Jobs over a share are deferred to the group's own queue and resumed by
ConcurrencyLimit::ResumeWorker. They are delayed, not dropped. A scoped job waits only behind queued jobs of its own group, or when the whole worker is at its limit.
The shared example a worker with a concurrency limit scope is the one introduced by !268561 (merged), added here unchanged so this MR does not depend on it. Whichever of the three wave MRs merges first carries the file; the others pick it up with no conflict.
References
- https://gitlab.com/groups/gitlab-org/-/work_items/24422 (parent, confidential)
- https://gitlab.com/gitlab-org/gitlab/-/work_items/637178, https://gitlab.com/gitlab-org/gitlab/-/work_items/636880, https://gitlab.com/gitlab-org/gitlab/-/work_items/636877, https://gitlab.com/gitlab-org/gitlab/-/work_items/636901 (per-worker issues, confidential)
- Rollout issue: #639186
- Wave one: !268561 (merged) and #638713
- Wave two: !268941 (merged) and #638936
- !268358 (closed) (the feature category change, included here)
- !266230 (merged) (free-slot admission, prerequisite for deleting the bypasses)
- #637499 (O(1) per-scope admission, prerequisite for enabling the scope)
- !226514 (merged) and !228036 (merged) (earlier limit raises for two of these workers)
Screenshots or screen recordings
No UI change.
How to set up and validate locally
-
Give the workers a limit above zero and enable the flags. In a Rails console:
Feature.enable(:sidekiq_concurrency_limit_default_calculation) Feature.enable(:sidekiq_concurrency_limit_scope) Feature.enable(:ci_pipeline_execution_concurrency_limit_scope_wave_three)Then restart Rails and Sidekiq with a fleet size, for example
GITLAB_SIDEKIQ_MAX_REPLICAS=1 SIDEKIQ_CONCURRENCY=10. -
Check the limits and the scope. With the fleet above,
Ci::ArchiveTraceWorkerandPipelineProcessWorkerget a limit of 6 (60% of 10 threads) andCi::ExecuteBuildHooksWorkerandCi::BuildTraceChunkFlushWorkerget 5 (45%, rounded up), and each top-level group gets 1 slot on any of them:Ci::ArchiveTraceWorker.get_concurrency_limit # => 6 Ci::ExecuteBuildHooksWorker.get_concurrency_limit # => 5 scope = Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(PipelineProcessWorker) scope.type # => :root_namespace scope.limit_for(6) # => 1 Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(Ci::BuildTraceChunkFlushWorker).limit_for(5) # => 1Scope.forreturnsnilwhile any of the flags is off, or after enabling the worker'sdisable_sidekiq_concurrency_limit_scope_<WorkerClass>kill switch. -
Run a pipeline with several jobs in one project and watch
log/sidekiq.logforconcurrency_limitdeferrals ofPipelineProcessWorkerwithconcurrency_limit_scopeset toroot_namespace:<group path>. Jobs from a project in a second top-level group start right away. -
bundle exec rake gitlab:sidekiq:all_queues_yml:checkpasses with the updatedapp/workers/all_queues.yml.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.