Loading
Limit concurrent pipeline creation jobs per user
What does this MR do and why?
Gives each user a fair share of the CreatePipelineWorker and MergeRequests::CreatePipelineWorker concurrency limits. Both workers declare concurrency_limit_scope :user, so jobs over the per-user limit are deferred to the user's own queue and resumed when the user's earlier jobs finish. Jobs are delayed, not dropped.
Details
CreatePipelineWorkercovers pipelines from Git pushes, the GraphQLpipelineCreatemutation withasync: true, and turning on Auto DevOps.MergeRequests::CreatePipelineWorkercovers merge request pipelines.- For this proof of concept, the limits come from environment variables, read when the worker loads.
0(the default) means no per-user limit. Set them for both Rails and Sidekiq, so the job is scoped the same way when it is enqueued and when it runs.CreatePipelineWorker:GITLAB_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USERMergeRequests::CreatePipelineWorker:GITLAB_MR_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER
- Each worker counts its own jobs, so a user can run up to the limit of each worker at the same time.
- Each worker's concurrency limit still caps its per-user limit. Neither worker sets
concurrency_limit, so the worker limit is the default one, calculated from the fleet size. When the worker limit is0, there is no per-user limit either. - The limit is enforced when the
sidekiq_concurrency_limit_scopeflag from !264227 (merged) is enabled, together with thesidekiq_concurrency_limit_middlewareops flag. There is no separate log-only step: enabling the scope flag defers jobs over the per-user limit. - A pipeline creation request is aged off after 5 minutes, so a job deferred for longer stops showing in
MergeRequest#pipeline_creating?. This already happens with the worker concurrency limit and with queue latency. !267879 (merged) keeps the request alive while the job is deferred.
References
Problem: https://gitlab.com/gitlab-org/gitlab/-/work_items/604640
| Step | Merge request | What it does | Needs |
|---|---|---|---|
| 1 | !264227 (merged) (merged) | Adds concurrency_limit_scope, a per-namespace or per-user share of a worker's concurrency limit. |
- |
| 2 | !264408 (merged) | Adds the on_concurrency_limit_deferred and on_concurrency_limit_resumed worker hooks. |
1 |
| 3 | This MR | Uses the per-user scope for CreatePipelineWorker and MergeRequests::CreatePipelineWorker. |
1 |
| 4 | !267879 (merged) | Keeps a pipeline creation request in progress while its job is deferred, using the hooks from step 2. | 2 |
Steps 3 and 4 do not depend on each other and can merge in any order.
How to set up and validate locally
- Start Rails and Sidekiq with
GITLAB_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER=1andGITLAB_MR_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER=1. For a worker limit above0, also setGITLAB_SIDEKIQ_MAX_REPLICASandSIDEKIQ_CONCURRENCY, for exampleGITLAB_SIDEKIQ_MAX_REPLICAS=1 SIDEKIQ_CONCURRENCY=20. - Enable the
sidekiq_concurrency_limit_middleware,sidekiq_concurrency_limit_default_calculationandsidekiq_concurrency_limit_scopefeature flags. - As one user, push to 3 branches of a project quickly.
- Check that only one
CreatePipelineWorkerjob runs at a time for that user, and the others are deferred and then resumed (concurrency_limitentries insidekiq.log). - As one user, push to 3 branches with open merge requests quickly, and check the same for
MergeRequests::CreatePipelineWorker.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
Edited by Furkan Ayhan