Loading
Use a share of the worker limit for pipeline creation per user
What does this MR do and why?
Replaces the per-user limit env vars of CreatePipelineWorker and MergeRequests::CreatePipelineWorker with a share of each worker's own concurrency limit, behind a new feature flag.
concurrency_limit_scope(:user, max_percentage: 0.3,
enabled: -> { Feature.enabled?(:ci_pipeline_creation_concurrency_limit_scope, :instance) })- Why a share: as discussed in !268561 (comment 4180431062), the goal is that one user cannot fill pipeline creation, not "at most N per user". A share follows shard resizes and urgency changes, and needs no env var or setting.
- Why 30%: both workers are high urgency, so their limit is 35% of the shard's threads. 30% of that caps one user at about 10% of the shard, the bound proposed in https://gitlab.com/gitlab-org/gitlab/-/work_items/604640. It is the same share !268561 (merged) uses for
Ci::DropPipelineWorker. - Nothing changes until two flags are on: the new
ci_pipeline_creation_concurrency_limit_scopeflag and the instance-widesidekiq_concurrency_limit_scope(#636157). One flag covers both workers. Each worker keeps its owndisable_sidekiq_concurrency_limit_scope_<WorkerClass>kill switch. - The env vars (
GITLAB_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER,GITLAB_MR_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER) are removed. They were never set on GitLab.com.
The rollout waits on #637499, as these are high-volume workers.
References
- https://gitlab.com/gitlab-org/gitlab/-/work_items/604640
- Rollout issue: #639567
- !264410 (merged) (added the env var limits)
- !268561 (merged) (same approach for Pipeline Execution workers)
Screenshots or screen recordings
No UI change.
How to set up and validate locally
-
Give the workers a limit above zero and enable the flags:
Feature.enable(:sidekiq_concurrency_limit_default_calculation) Feature.enable(:sidekiq_concurrency_limit_scope) Feature.enable(:ci_pipeline_creation_concurrency_limit_scope)Then restart Rails and Sidekiq with
GITLAB_SIDEKIQ_MAX_REPLICAS=1 SIDEKIQ_CONCURRENCY=20. -
Check the scope. The worker limit is 7 (35% of 20 threads), so each user gets 3 slots (30% of 7, rounded up):
scope = Gitlab::SidekiqMiddleware::ConcurrencyLimit::Scope.for(CreatePipelineWorker) scope.type # => :user scope.limit_for(7) # => 3Scope.forreturnsnilwhile any of the flags is off.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
Edited by Furkan Ayhan