Keep pipeline creation requests alive while deferred
What does this MR do and why?
Keeps a pipeline creation request in progress while its CreatePipelineWorker or MergeRequests::CreatePipelineWorker job is deferred by the concurrency limit.
Today a request is aged off after 5 minutes. A deferred job can wait for hours, so MergeRequest#pipeline_creating? stops reporting the pending creation, and the MR shows "The pipeline must succeed." as if CI failed, instead of waiting for the pipeline. This happens already with the worker concurrency limit, and gets more likely with the per-user limit of !264410 (merged).
This uses the hooks from !264408 (merged):
on_concurrency_limit_deferred: keeps the request for 1 hour, and restores it if it expired before the job was deferred.on_concurrency_limit_resumed: keeps it for at least 5 more minutes, and restores it if it expired while the job waited.
Details
- A kept-alive request stores
kept_alive_untilin its entry and counts as in progress until then, past the usual 5 minute cutoff onstarted_at.started_atis unchanged, so thecreate_pipeline_from_commitSLI is not affected. - Writes never shorten a request:
- Its Redis key's expiry, because a key can hold the requests of several jobs:
EXPIRE NXsets it on a new key, thenEXPIRE GTonly extends it. First use ofEXPIRE … NX/GTin the codebase. Needs Redis 7.0, which is the minimum supported version. - Its
kept_alive_until, so a resume that fails and leaves the job deferred does not cut the 1 hour deadline to 5 minutes.
- Its Redis key's expiry, because a key can hold the requests of several jobs:
- Because a key can now live longer, merge request requests are aged by their entry, the same way ref requests already are. Otherwise a request of a lost job would stay in progress while another request keeps the key alive. This applies to all merge request requests, not only deferred ones.
- A request that expired is restored: for 1 hour when the job is deferred, for 5 minutes when it is resumed, as a resume can be retried after the job ran and finished a ref request.
- Trade-offs:
- A job that waits in the queue for over 1 hour behaves as on
masterfor the rest of the wait. Its request is restored when the job is resumed, or kept for 1 more hour if the job is deferred again. - A deferred job that is dropped from the concurrency limit queue (for example with
drop_jobs!) leaves its request in progress for up to 1 hour. - Merge request and project request keys can keep final statuses, which GraphQL exposes, for up to 1 hour after a deferral instead of 5 minutes.
- A job that waits in the queue for over 1 hour behaves as on
- Not behind a feature flag. The hooks only run when the concurrency limit defers a job, which is behind the
sidekiq_concurrency_limit_middlewareops flag. The expiry and aging changes apply to all requests.
References
Problem: https://gitlab.com/gitlab-org/gitlab/-/work_items/604640
| Step | Merge request | What it does | Needs |
|---|---|---|---|
| 1 | !264227 (merged) (merged) | Adds concurrency_limit_scope, a per-namespace or per-user share of a worker's concurrency limit. |
- |
| 2 | !264408 (merged) (merged) | Adds the on_concurrency_limit_deferred and on_concurrency_limit_resumed worker hooks. |
1 |
| 3 | !264410 (merged) (merged) | Uses the per-user scope for CreatePipelineWorker and MergeRequests::CreatePipelineWorker. |
1 |
| 4 | This MR | Keeps a pipeline creation request in progress while its job is deferred, using the hooks from step 2. | 2 |
Screenshots or screen recordings
No UI change. The diagrams show a new push to an MR in a project with "Pipelines must succeed" turned on.
Before
sequenceDiagram
actor User
participant MR as Merge request
participant Req as Pipeline creation request
participant CL as Concurrency limit
participant W as Pipeline creation job
User->>MR: Push
MR->>Req: start_for_merge_request (in progress)
MR->>CL: Enqueue job
CL->>CL: Limit reached, job deferred
Note over MR: Status: ci_still_running
Note over Req: After 5 minutes, the request ages off
MR->>Req: pipeline_creating?
Req-->>MR: false
Note over MR: Status: ci_must_pass ("The pipeline must succeed."), as if CI failed
CL->>W: Resume, maybe hours later
W->>MR: Pipeline created
Note over MR: Status: ci_still_running againAfter
sequenceDiagram
actor User
participant MR as Merge request
participant Req as Pipeline creation request
participant CL as Concurrency limit
participant W as Pipeline creation job
User->>MR: Push
MR->>Req: start_for_merge_request (in progress)
MR->>CL: Enqueue job
CL->>CL: Limit reached, job deferred
CL->>Req: on_concurrency_limit_deferred: keep alive for 1 hour
Note over MR: Status: ci_still_running
Note over Req: After 5 minutes, still in progress
MR->>Req: pipeline_creating?
Req-->>MR: true
Note over MR: Status: ci_still_running
CL->>Req: on_concurrency_limit_resumed: keep alive for at least 5 more minutes
CL->>W: Resume
W->>MR: Pipeline created
Note over MR: Status: ci_still_runningHow to set up and validate locally
-
In a Rails console, start a request and call the deferred hook:
mr = MergeRequest.last request = Ci::PipelineCreation::Requests.start_for_merge_request(mr) args = [mr.project_id, mr.author_id, mr.id, { 'pipeline_creation_request' => request }] MergeRequests::CreatePipelineWorker.on_concurrency_limit_deferred(args) Time.at(Ci::PipelineCreation::Requests.hget(request)['kept_alive_until']) # about 1 hour from now Gitlab::Redis::SharedState.with { |r| r.ttl(request['key']) } # about 3600 -
Simulate the request expiring while the job waits, then call the resumed hook:
Gitlab::Redis::SharedState.with { |r| r.del(request['key']) } MergeRequests::CreatePipelineWorker.on_concurrency_limit_resumed(args) Ci::PipelineCreation::Requests.hget(request)['status'] # "in_progress" mr.pipeline_creating? # true
End to end test with a deferred job
In GDK the worker limit is 0, so the per-user limit has no effect until a worker limit is set. In env.runit:
export GITLAB_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER=1
export GITLAB_MR_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER=1
export GITLAB_SIDEKIQ_MAX_REPLICAS=1
export SIDEKIQ_CONCURRENCY=20Feature.enable(:sidekiq_concurrency_limit_scope)
Feature.enable(:sidekiq_concurrency_limit_default_calculation)Then:
- Create a project with "Pipelines must succeed" turned on, and two MRs, A and B, by the same user.
- Make pipeline creation slow, for example by adding a
sleeptoperform, and push to A. A's job takes the user's only slot. - Remove the
sleepand push to B. B's job is deferred. - Check B's
detailedMergeStatuswhile it waits.
| B's age | master |
This MR |
|---|---|---|
| Under 5 minutes | CI_STILL_RUNNING |
CI_STILL_RUNNING |
| Over 5 minutes, still deferred | CI_MUST_PASS, no pipeline |
CI_STILL_RUNNING |
| Resumed, pipeline created | MERGEABLE |
MERGEABLE |
This MR gave the same results with track_ref_pipeline_creation off (merge request requests only) and on (merge request and ref requests).
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.