Limit concurrent pipeline creation jobs per user

What does this MR do and why?

Gives each user a fair share of the CreatePipelineWorker and MergeRequests::CreatePipelineWorker concurrency limits. Both workers declare concurrency_limit_scope :user, so jobs over the per-user limit are deferred to the user's own queue and resumed when the user's earlier jobs finish. Jobs are delayed, not dropped.

Details
  • CreatePipelineWorker covers pipelines from Git pushes, the GraphQL pipelineCreate mutation with async: true, and turning on Auto DevOps.
  • MergeRequests::CreatePipelineWorker covers merge request pipelines.
  • For this proof of concept, the limits come from environment variables, read when the worker loads. 0 (the default) means no per-user limit. Set them for both Rails and Sidekiq, so the job is scoped the same way when it is enqueued and when it runs.
    • CreatePipelineWorker: GITLAB_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER
    • MergeRequests::CreatePipelineWorker: GITLAB_MR_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER
  • Each worker counts its own jobs, so a user can run up to the limit of each worker at the same time.
  • Each worker's concurrency limit still caps its per-user limit. Neither worker sets concurrency_limit, so the worker limit is the default one, calculated from the fleet size. When the worker limit is 0, there is no per-user limit either.
  • The limit is enforced when the sidekiq_concurrency_limit_scope flag from !264227 (merged) is enabled, together with the sidekiq_concurrency_limit_middleware ops flag. There is no separate log-only step: enabling the scope flag defers jobs over the per-user limit.
  • A pipeline creation request is aged off after 5 minutes, so a job deferred for longer stops showing in MergeRequest#pipeline_creating?. This already happens with the worker concurrency limit and with queue latency. !267879 (merged) keeps the request alive while the job is deferred.

References

Problem: https://gitlab.com/gitlab-org/gitlab/-/work_items/604640

Step Merge request What it does Needs
1 !264227 (merged) (merged) Adds concurrency_limit_scope, a per-namespace or per-user share of a worker's concurrency limit. -
2 !264408 (merged) Adds the on_concurrency_limit_deferred and on_concurrency_limit_resumed worker hooks. 1
3 This MR Uses the per-user scope for CreatePipelineWorker and MergeRequests::CreatePipelineWorker. 1
4 !267879 (merged) Keeps a pipeline creation request in progress while its job is deferred, using the hooks from step 2. 2

Steps 3 and 4 do not depend on each other and can merge in any order.

How to set up and validate locally

  1. Start Rails and Sidekiq with GITLAB_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER=1 and GITLAB_MR_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER=1. For a worker limit above 0, also set GITLAB_SIDEKIQ_MAX_REPLICAS and SIDEKIQ_CONCURRENCY, for example GITLAB_SIDEKIQ_MAX_REPLICAS=1 SIDEKIQ_CONCURRENCY=20.
  2. Enable the sidekiq_concurrency_limit_middleware, sidekiq_concurrency_limit_default_calculation and sidekiq_concurrency_limit_scope feature flags.
  3. As one user, push to 3 branches of a project quickly.
  4. Check that only one CreatePipelineWorker job runs at a time for that user, and the others are deferred and then resumed (concurrency_limit entries in sidekiq.log).
  5. As one user, push to 3 branches with open merge requests quickly, and check the same for MergeRequests::CreatePipelineWorker.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Furkan Ayhan

Merge request reports

Loading
Loading