Protect a pipeline when a non-interruptible job runs

Marius Bobinrequested to merge
621543-protect-pipeline into master

What does this MR do and why?

A job that cannot be interrupted takes its pipeline out of reach of auto-cancellation. This MR records that fact when the job starts.

graph LR
    B["Ci::Build<br/>pending => running"] -->|"in the request"| C["CandidateCache#protect"]
    C --> R["redis<br/>...:protected"]
    B -->|"behind it"| W["ProtectPipelineWorker"]
    W --> D["p_ci_pipeline_processing_data<br/>interruptible_protected"]

Two stores hold the answer. Redis is written inside the request that starts the job, so it answers at once. A worker writes a Postgres column behind it, which outlives the cache. A pipeline loses its protection only if both stores miss the fact.

The redis write adds no database query. The runner already makes more than 100 queries to pick up a job, so that path cannot take another one.

The answer is held in a second sorted set, rather than by removing the pipeline from the candidate set. A pipeline whose auto_cancel_on_new_commit is interruptible must still be cancelled, and only its interruptible jobs get cancelled. Removing it from the candidates would stop it being cancelled at all. Holding the answer apart lets a newer pipeline apply the mode when it reads, where the policy is already loaded.

Nothing reads the answer yet. The reader arrives in a later merge request.

Everything is behind the feature flag ci_redundant_pipeline_candidates_cache, type wip, disabled by default.

Database

Ci::PipelineProcessingData.protect writes one row per pipeline. It runs when a job that cannot be interrupted starts, from a worker keyed on the pipeline, so every job of one pipeline collapses into one write.

INSERT INTO "p_ci_pipeline_processing_data"
  ("pipeline_id","partition_id","project_id","interruptible_protected")
VALUES (730, 100, 42, TRUE)
ON CONFLICT ("pipeline_id","partition_id") DO UPDATE SET
  interruptible_protected = true
  WHERE NOT p_ci_pipeline_processing_data.interruptible_protected

The plan comes from a local database where the table is empty, so it shows the conflict arbiter rather than row counts:

Insert on p_ci_pipeline_processing_data  (cost=0.00..0.01 rows=0 width=0)
  Conflict Resolution: UPDATE
  Conflict Arbiter Indexes: p_ci_pipeline_processing_data_pkey
  Conflict Filter: (NOT p_ci_pipeline_processing_data.interruptible_protected)

The conflict predicate stops a repeated job start from rewriting a row that already holds the answer.

database

How to set up and validate locally

bundle exec rspec spec/lib/gitlab/ci/redundant_pipelines/candidate_cache_spec.rb \
                  spec/models/ci/pipeline_processing_data_spec.rb \
                  spec/workers/ci/redundant_pipelines/protect_pipeline_worker_spec.rb \
                  spec/models/ci/build_spec.rb

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Marius Bobin

Merge request reports