Keep pipeline creation requests alive while deferred

What does this MR do and why?

Keeps a pipeline creation request in progress while its CreatePipelineWorker or MergeRequests::CreatePipelineWorker job is deferred by the concurrency limit.

Today a request is aged off after 5 minutes. A deferred job can wait for hours, so MergeRequest#pipeline_creating? stops reporting the pending creation, and the MR shows "The pipeline must succeed." as if CI failed, instead of waiting for the pipeline. This happens already with the worker concurrency limit, and gets more likely with the per-user limit of !264410 (merged).

This uses the hooks from !264408 (merged):

  • on_concurrency_limit_deferred: keeps the request for 1 hour, and restores it if it expired before the job was deferred.
  • on_concurrency_limit_resumed: keeps it for at least 5 more minutes, and restores it if it expired while the job waited.
Details
  • A kept-alive request stores kept_alive_until in its entry and counts as in progress until then, past the usual 5 minute cutoff on started_at. started_at is unchanged, so the create_pipeline_from_commit SLI is not affected.
  • Writes never shorten a request:
    • Its Redis key's expiry, because a key can hold the requests of several jobs: EXPIRE NX sets it on a new key, then EXPIRE GT only extends it. First use of EXPIRE … NX/GT in the codebase. Needs Redis 7.0, which is the minimum supported version.
    • Its kept_alive_until, so a resume that fails and leaves the job deferred does not cut the 1 hour deadline to 5 minutes.
  • Because a key can now live longer, merge request requests are aged by their entry, the same way ref requests already are. Otherwise a request of a lost job would stay in progress while another request keeps the key alive. This applies to all merge request requests, not only deferred ones.
  • A request that expired is restored: for 1 hour when the job is deferred, for 5 minutes when it is resumed, as a resume can be retried after the job ran and finished a ref request.
  • Trade-offs:
    • A job that waits in the queue for over 1 hour behaves as on master for the rest of the wait. Its request is restored when the job is resumed, or kept for 1 more hour if the job is deferred again.
    • A deferred job that is dropped from the concurrency limit queue (for example with drop_jobs!) leaves its request in progress for up to 1 hour.
    • Merge request and project request keys can keep final statuses, which GraphQL exposes, for up to 1 hour after a deferral instead of 5 minutes.
  • Not behind a feature flag. The hooks only run when the concurrency limit defers a job, which is behind the sidekiq_concurrency_limit_middleware ops flag. The expiry and aging changes apply to all requests.

References

Problem: https://gitlab.com/gitlab-org/gitlab/-/work_items/604640

Step Merge request What it does Needs
1 !264227 (merged) (merged) Adds concurrency_limit_scope, a per-namespace or per-user share of a worker's concurrency limit. -
2 !264408 (merged) (merged) Adds the on_concurrency_limit_deferred and on_concurrency_limit_resumed worker hooks. 1
3 !264410 (merged) (merged) Uses the per-user scope for CreatePipelineWorker and MergeRequests::CreatePipelineWorker. 1
4 This MR Keeps a pipeline creation request in progress while its job is deferred, using the hooks from step 2. 2

Screenshots or screen recordings

No UI change. The diagrams show a new push to an MR in a project with "Pipelines must succeed" turned on.

Before

sequenceDiagram
    actor User
    participant MR as Merge request
    participant Req as Pipeline creation request
    participant CL as Concurrency limit
    participant W as Pipeline creation job

    User->>MR: Push
    MR->>Req: start_for_merge_request (in progress)
    MR->>CL: Enqueue job
    CL->>CL: Limit reached, job deferred
    Note over MR: Status: ci_still_running
    Note over Req: After 5 minutes, the request ages off
    MR->>Req: pipeline_creating?
    Req-->>MR: false
    Note over MR: Status: ci_must_pass ("The pipeline must succeed."), as if CI failed
    CL->>W: Resume, maybe hours later
    W->>MR: Pipeline created
    Note over MR: Status: ci_still_running again

After

sequenceDiagram
    actor User
    participant MR as Merge request
    participant Req as Pipeline creation request
    participant CL as Concurrency limit
    participant W as Pipeline creation job

    User->>MR: Push
    MR->>Req: start_for_merge_request (in progress)
    MR->>CL: Enqueue job
    CL->>CL: Limit reached, job deferred
    CL->>Req: on_concurrency_limit_deferred: keep alive for 1 hour
    Note over MR: Status: ci_still_running
    Note over Req: After 5 minutes, still in progress
    MR->>Req: pipeline_creating?
    Req-->>MR: true
    Note over MR: Status: ci_still_running
    CL->>Req: on_concurrency_limit_resumed: keep alive for at least 5 more minutes
    CL->>W: Resume
    W->>MR: Pipeline created
    Note over MR: Status: ci_still_running

How to set up and validate locally

  1. In a Rails console, start a request and call the deferred hook:

    mr = MergeRequest.last
    request = Ci::PipelineCreation::Requests.start_for_merge_request(mr)
    args = [mr.project_id, mr.author_id, mr.id, { 'pipeline_creation_request' => request }]
    
    MergeRequests::CreatePipelineWorker.on_concurrency_limit_deferred(args)
    
    Time.at(Ci::PipelineCreation::Requests.hget(request)['kept_alive_until']) # about 1 hour from now
    Gitlab::Redis::SharedState.with { |r| r.ttl(request['key']) } # about 3600
  2. Simulate the request expiring while the job waits, then call the resumed hook:

    Gitlab::Redis::SharedState.with { |r| r.del(request['key']) }
    
    MergeRequests::CreatePipelineWorker.on_concurrency_limit_resumed(args)
    
    Ci::PipelineCreation::Requests.hget(request)['status'] # "in_progress"
    mr.pipeline_creating? # true
End to end test with a deferred job

In GDK the worker limit is 0, so the per-user limit has no effect until a worker limit is set. In env.runit:

export GITLAB_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER=1
export GITLAB_MR_PIPELINE_CREATION_CONCURRENCY_LIMIT_PER_USER=1
export GITLAB_SIDEKIQ_MAX_REPLICAS=1
export SIDEKIQ_CONCURRENCY=20
Feature.enable(:sidekiq_concurrency_limit_scope)
Feature.enable(:sidekiq_concurrency_limit_default_calculation)

Then:

  1. Create a project with "Pipelines must succeed" turned on, and two MRs, A and B, by the same user.
  2. Make pipeline creation slow, for example by adding a sleep to perform, and push to A. A's job takes the user's only slot.
  3. Remove the sleep and push to B. B's job is deferred.
  4. Check B's detailedMergeStatus while it waits.
B's age master This MR
Under 5 minutes CI_STILL_RUNNING CI_STILL_RUNNING
Over 5 minutes, still deferred CI_MUST_PASS, no pipeline CI_STILL_RUNNING
Resumed, pipeline created MERGEABLE MERGEABLE

This MR gave the same results with track_ref_pipeline_creation off (merge request requests only) and on (merge request and ref requests).

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Furkan Ayhan

Merge request reports

Loading
Loading