Add concurrency limit scope to pipeline execution wave one workers

What does this MR do and why?

Declares concurrency_limit_scope (!264227 (merged)) on three Pipeline Execution workers, so one top-level group or user is capped at a share of each worker's Sidekiq concurrency limit instead of being able to fill it:

Worker Scope key Share of the worker limit
Ci::CreateDownstreamPipelineWorker top-level group 40%
MergeTrains::RefreshWorker top-level group 40%
Ci::DropPipelineWorker user 30%
Ci::DropPipelineForBlockedUserWorker (inherits it from Ci::DropPipelineWorker) user 30%
  • Nothing changes until two flags are on. The scope applies only while the instance-wide sidekiq_concurrency_limit_scope flag (#636157, not enabled anywhere yet) and the new ci_pipeline_execution_concurrency_limit_scope_wave_one flag (rollout: #638713) are both enabled. One flag covers the three workers. Each worker keeps its own disable_sidekiq_concurrency_limit_scope_<WorkerClass> kill switch, so they can still be enabled or stopped one at a time.
  • A share, not an absolute number. max_percentage: is a share of the worker's own limit, which on GitLab.com is derived from the shard's capacity (25% of the shard for low urgency, 35% for high). 40% of the low-urgency default and 30% of the high-urgency default both come to about 10% of the shard per tenant, the bound Hordur Freyr Yngvason proposed in https://gitlab.com/gitlab-org/gitlab/-/work_items/604640 ("up to 10% of total capacity"). No environment variable or setting is needed, and the share follows shard resizes.
  • Keys follow the enqueue context. Downstream pipeline creation and merge train refreshes are enqueued under the project's context, so they key on the top-level group. Pipeline drops are enqueued with bulk_perform_async_with_contexts(..., context_proc: -> { { user: context_user } }), where the context user is the banned or blocked user, so they key on that user, not on whoever triggered the drop. Jobs with no tenant in context (for example MergeTrains::RefreshWorker jobs from the unstick cron) stay under the worker limit only and are counted in sidekiq_concurrency_limit_scope_unattributed_jobs_total.
  • Ci::DropPipelineForBlockedUserWorker inherits the scope. It subclasses Ci::DropPipelineWorker, and worker attributes are inherited through Gitlab::ClassAttributes, so the user-block path is scoped too: 30% of its low-urgency limit, about 7.5% of the shard, per blocked user. It has its own disable_sidekiq_concurrency_limit_scope_Ci::DropPipelineForBlockedUserWorker kill switch.
  • Jobs over a share are deferred to the tenant's own queue and resumed by ConcurrencyLimit::ResumeWorker. They are delayed, not dropped.

These three workers run at low volume on GitLab.com. #637499 asks that only low-volume workers adopt the scope, one at a time, until scoped admission is O(1). The rollout issue enables them one at a time through the per-worker kill switches.

References

Screenshots or screen recordings

No UI change.

How to set up and validate locally

  1. Give the workers a limit above zero and enable the flags. In a Rails console:

    Feature.enable(:sidekiq_concurrency_limit_default_calculation)
    Feature.enable(:sidekiq_concurrency_limit_scope)
    Feature.enable(:ci_pipeline_execution_concurrency_limit_scope_wave_one)

    Then restart Rails and Sidekiq with a fleet size, for example GITLAB_SIDEKIQ_MAX_REPLICAS=1 SIDEKIQ_CONCURRENCY=10.

  2. Check that the scope resolves. With the fleet above, the worker's own limit is 3 (25% of 10 threads, rounded up), so each top-level group gets 2 slots:

    scope = Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(Ci::CreateDownstreamPipelineWorker)
    scope.type         # => :root_namespace
    scope.limit_for(3) # => 2
    Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(Ci::DropPipelineWorker).type # => :user

    Scope.for returns nil while any of the flags is off, or after enabling the worker's disable_sidekiq_concurrency_limit_scope_<WorkerClass> kill switch.

  3. Trigger more downstream pipelines from one project than the share allows, and watch log/sidekiq.log for concurrency_limit deferrals with concurrency_limit_scope set to root_namespace:<group path>. Jobs from a second top-level group start right away.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Max Fan

Merge request reports

Loading
Loading