Add concurrency limit scope to pipeline execution wave three workers

What does this MR do and why?

Third wave of the per-tenant concurrency limit scope for Pipeline Execution workers, covering the four highest-volume ones. Three changes:

Worker GitLab.com shard Worker limit (share of the shard) Scope key Share of the worker limit
PipelineProcessWorker urgent-ci-pipeline 60% (unchanged) top-level group 15%
Ci::BuildTraceChunkFlushWorker urgent-other 45% (unchanged) top-level group 20%
Ci::ArchiveTraceWorker low-urgency-ci-pipeline 32% -> 60% top-level group 15%
Ci::ExecuteBuildHooksWorker catchall-b -> low-urgency-ci-pipeline 25% default -> 45% top-level group 20%
  1. Declares concurrency_limit_scope (!264227 (merged)) on the four workers, so one top-level group is capped at a share of each worker's Sidekiq concurrency limit instead of being able to fill it. Same attribute and share rule as !268561 (merged) and !268941 (merged), behind a new flag.
  2. Raises two worker limits. Ci::ArchiveTraceWorker goes from 32% to 60% of the shard and Ci::ExecuteBuildHooksWorker gets 45% instead of the 25% low-urgency default. Both limits sit below the workers' GitLab.com peaks, which is why both workers run with the limiter bypassed there today. 60% and 45% are the values !226514 (merged) and !228036 (merged) picked for PipelineProcessWorker and Ci::BuildTraceChunkFlushWorker for the same reason, and both leave the same headroom over the past week's peak as those two workers have today. The other two limits already clear their peaks and stay as they are.
  3. Moves Ci::ExecuteBuildHooksWorker to the continuous_integration feature category. This is the commit from !268358 (closed), cherry-picked with its author kept, because Pipeline Execution is taking the worker over and the limit above has to be sized for the shard it will run on. On GitLab.com the routing rules send continuous_integration + low urgency to low-urgency-ci-pipeline, so this moves the worker off catchall-b.

What each change needs before it does anything on GitLab.com:

  • The scope needs two flags. The instance-wide sidekiq_concurrency_limit_scope flag (#636157) and the new ci_pipeline_execution_concurrency_limit_scope_wave_three flag (rollout: #639186). One flag covers the four workers; each keeps its own disable_sidekiq_concurrency_limit_scope_<WorkerClass> kill switch so they can be enabled or stopped one at a time. Enabling in production waits for #637499 part 1 (O(1) per-scope admission), as for wave two.
  • The limits need the bypass flags deleted. All four workers have disable_sidekiq_concurrency_limit_middleware_<WorkerClass> enabled on GitLab.com, which makes the middleware skip them entirely, so the new percentages are a no-op for enforcement until the rollout issue deletes those flags. The sidekiq_concurrency_limit_max_concurrent_jobs and sidekiq_concurrency_limit_current_limit gauges for the two workers do move on the deploy, because ConcurrencyLimitSampler reads the configured limit without checking the bypass. That happens per worker, after !266230 (merged) (free-slot admission) is deployed and enabled for the worker, since the non-empty-list latch is what put these workers behind the bypass in the first place.
  • The shard move happens on deploy. It is a routing consequence of the category change, not a flag. The rollout issue lists low-urgency-ci-pipeline saturation as the first thing to watch.

Other notes:

  • A share, not an absolute number. max_percentage: is a share of the worker's own limit, which on GitLab.com is derived from the shard's capacity. 15% of a 60% limit and 20% of a 45% limit both come to 9% of the shard per top-level group, about the 10% bound waves one and two used. The share follows shard resizes.
  • Keys follow the enqueue context. All four workers are enqueued under the project's context (pipeline and job status transitions, trace chunk writes, the job-finished hand-off), so they key on the top-level group. Jobs with no top-level group in context, such as the cron-driven ones, stay under the worker limit only and are counted in sidekiq_concurrency_limit_scope_unattributed_jobs_total.
  • Jobs over a share are deferred to the group's own queue and resumed by ConcurrencyLimit::ResumeWorker. They are delayed, not dropped. A scoped job waits only behind queued jobs of its own group, or when the whole worker is at its limit.

The shared example a worker with a concurrency limit scope is the one introduced by !268561 (merged), added here unchanged so this MR does not depend on it. Whichever of the three wave MRs merges first carries the file; the others pick it up with no conflict.

References

Screenshots or screen recordings

No UI change.

How to set up and validate locally

  1. Give the workers a limit above zero and enable the flags. In a Rails console:

    Feature.enable(:sidekiq_concurrency_limit_default_calculation)
    Feature.enable(:sidekiq_concurrency_limit_scope)
    Feature.enable(:ci_pipeline_execution_concurrency_limit_scope_wave_three)

    Then restart Rails and Sidekiq with a fleet size, for example GITLAB_SIDEKIQ_MAX_REPLICAS=1 SIDEKIQ_CONCURRENCY=10.

  2. Check the limits and the scope. With the fleet above, Ci::ArchiveTraceWorker and PipelineProcessWorker get a limit of 6 (60% of 10 threads) and Ci::ExecuteBuildHooksWorker and Ci::BuildTraceChunkFlushWorker get 5 (45%, rounded up), and each top-level group gets 1 slot on any of them:

    Ci::ArchiveTraceWorker.get_concurrency_limit        # => 6
    Ci::ExecuteBuildHooksWorker.get_concurrency_limit   # => 5
    scope = Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(PipelineProcessWorker)
    scope.type         # => :root_namespace
    scope.limit_for(6) # => 1
    Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(Ci::BuildTraceChunkFlushWorker).limit_for(5) # => 1

    Scope.for returns nil while any of the flags is off, or after enabling the worker's disable_sidekiq_concurrency_limit_scope_<WorkerClass> kill switch.

  3. Run a pipeline with several jobs in one project and watch log/sidekiq.log for concurrency_limit deferrals of PipelineProcessWorker with concurrency_limit_scope set to root_namespace:<group path>. Jobs from a project in a second top-level group start right away.

  4. bundle exec rake gitlab:sidekiq:all_queues_yml:check passes with the updated app/workers/all_queues.yml.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Max Fan

Merge request reports

Loading
Loading