Cancel the older pipelines a newer pipeline replaces

What does this MR do and why?

When a newer pipeline starts, it takes the older pipelines of its ref out of Gitlab::Ci::RedundantPipelines::CandidateCache. This MR adds Ci::RedundantPipelines::CancelOlderPipelinesService to do that step and cancel those older pipelines.

graph LR
    P["newer pipeline"] --> O["CancelOlderPipelinesService"]
    O -->|"one redis call,<br/>at most 100"| C["CandidateCache<br/>take"]
    C -->|"each older pipeline"| S["CancelReplacedPipelineService"]
    S --> R["canceled, put back,<br/>or dropped"]

The service gets up to 100 older pipelines from the cache in one redis call. It sends each one to Ci::RedundantPipelines::CancelReplacedPipelineService, which is already on master. That service cancels the pipeline, puts it back in the cache, or drops it. The new service returns the number of pipelines it canceled.

The same call that takes a pipeline also removes it from the cache. Because of this, two newer pipelines cannot both cancel the same older pipeline.

The old path is Ci::PipelineCreation::CancelRedundantPipelinesService. This service does not run two of its queries:

• The search in p_ci_pipelines by project and ref.
• The check in p_ci_builds for jobs that cannot be interrupted.

The specs check that neither query runs.

The service looks up the commit at the tip of the ref only once for all the older pipelines. It does not look it up again for each pipeline.

Redis cost:

• One call to take the older pipelines.
• At most one ZSCORE for each older pipeline. This happens only when the auto-cancel policy checks protection in conservative mode and the column is not written yet.

The spec checks that 5 pipelines use 1 EVALSHA and 5 ZSCORE calls.

If auto-cancel does not apply to the newer pipeline, the service does nothing. If the cache is gone, it cancels nothing and runs no search.

Nothing calls CancelOlderPipelinesService yet. The worker change that calls it comes in a later merge request.

Everything is behind the feature flag ci_redundant_pipeline_candidates_cache. Its type is wip, and it is disabled by default.

How to set up and validate locally

bundle exec rspec spec/services/ci/redundant_pipelines/cancel_older_pipelines_service_spec.rb

References

• Discussion issue: #621543
• Parent proof-of-concept MR: !253751
• Earlier pieces of the same stack: !257741 (merged) and !259960 (merged)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.