Cancel redundant pipelines from the cache behind a flag

Marius Bobinrequested to merge
621543-poc-redis-redundant-pipeline-candidates into 621543-cancel-older-pipelines

What does this MR do and why?

Ci::CancelRedundantPipelinesWorker now picks one of two paths to cancel older pipelines. The new feature flag ci_redundant_pipeline_cancel_from_cache controls the choice.

graph LR
    W["CancelRedundantPipelinesWorker"] -->|"flag on"| C["CancelOlderPipelinesService<br/>take from the cache"]
    W -->|"flag off"| S["CancelRedundantPipelinesService<br/>search p_ci_pipelines"]

When the flag is on, the worker calls Ci::RedundantPipelines::CancelOlderPipelinesService from !260250. That service takes the older pipelines from the cache. When the flag is off, the worker calls the existing Ci::PipelineCreation::CancelRedundantPipelinesService, which searches p_ci_pipelines.

Two flags control the rollout:

ci_redundant_pipeline_candidates_cache   fills the cache (already on master)
ci_redundant_pipeline_cancel_from_cache  cancels from it (new here)

Enable the first flag, wait a couple of days, then enable the second. The wait is necessary. Without it, the first pipeline cancelled from the cache could be one that started before the cache existed. The cache never got that pipeline's older pipelines or its non-interruptible jobs. A newer commit would then cancel a pipeline that it must leave alone.

If something goes wrong, turn off the second flag. The worker goes back to the search, and the cache stays full.

If the cache is gone, the cache path cancels nothing. It does not go back to the search. This means a lost cache never puts the database back on this path. Cancellation is best-effort.

Flags are on by default in tests. For this reason, the auto-cancel examples in create_pipeline_service_spec.rb now pin the search path they were written for. One of these examples finishes a non-interruptible build without running it. The search path treats that build as protection for the pipeline. The cache does not, because it records protection only when a job starts to run.

A new integration spec, spec/services/ci/create_pipeline_service_redundant_pipelines_spec.rb, runs the real Ci::CreatePipelineService and workers with :sidekiq_inline. It covers:

  • Repeated pipelines for the commit that the ref points at.
  • A child pipeline that runs longer than its parent.
  • A child pipeline that its own configuration or its own protection keeps safe.
  • Protection that is only in redis before the column is written.
  • A retried pipeline.
  • A lost cache that cancels nothing.
  • In every flag state, only the creation chain asks for the cancellation.

The spec also checks that no project-and-ref search runs against p_ci_pipelines. The worker spec covers both flag states.

The flag ci_redundant_pipeline_cancel_from_cache is type gitlab_com_derisk and disabled by default. Its actor is the project. Rollout issue: #632587.

Database

This MR adds no new queries. When the flag is on, the worker no longer runs the p_ci_pipelines search or the p_ci_builds non-interruptible check.

How to set up and validate locally

bundle exec rspec spec/services/ci/create_pipeline_service_redundant_pipelines_spec.rb spec/workers/ci/cancel_redundant_pipelines_worker_spec.rb spec/services/ci/create_pipeline_service_spec.rb
Manual check

  1. Enable both flags for a project:

    project = Project.find_by_full_path('<path>')
Feature.enable(:ci_redundant_pipeline_candidates_cache, project)
Feature.enable(:ci_redundant_pipeline_cancel_from_cache, project)

  2. Add this .gitlab-ci.yml:

    job:
  interruptible: true
  script: sleep 600

  3. Push a commit. Wait until the pipeline is running.

  4. Push another commit to the same branch.

  5. Make sure that the older pipeline is canceled.

ci_redundant_pipeline_candidates_cache ci_redundant_pipeline_cancel_from_cache Cache Result
on on full Older pipeline canceled from the cache
on off full Older pipeline canceled by the search
on on flushed Nothing canceled

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Marius Bobin

Merge request reports