Improving canceling redundant pipelines with `interruptible`
Problem
Feature: Auto-cancel redundant pipelines.
The current behavior
The default of interruptible
is false for jobs. This suggests that all jobs are uninterruptible unless the default
section or a specific job in .gitlab-ci.yml
is set to interruptible: true
.
There is a scenario where, currently, this is not true, a pipeline can still be auto-canceled when no job is started.
As an example;
# https://gitlab.com/furkanayhan/test-project/-/blob/7ddf44269d7c310b030338d364124d81b6c3d1c9/.gitlab-ci.yml
test1:
tags: [jahdkjashd] # invalid tag to keep this pending
script: sleep 60
test2:
tags: [jahdkjashd] # invalid tag to keep this pending
script: sleep 60
When I send another commit to the same branch, these jobs will be canceled even if they don't have interruptible: true
;
https://gitlab.com/furkanayhan/test-project/-/pipelines/878843139
This is a documented behavior;
To completely cancel a running pipeline, all jobs must have
interruptible: true
, orinterruptible: false
jobs must not have started.
Its effect on downstream pipelines
Even though the behavior above can be expected behavior because of this sentence "Jobs that have not started yet (pending) are considered interruptible and safe to be cancelled.", it may raise unwanted behaviors for downstream pipelines for some users.
Steps and scenario:
- Let's say we only have two Runners. (In your runner's
config.toml
, setconcurrent = 2
in the global section) - Enable Auto-cancel redundant pipelines on your project
- Add a basic child pipeline in a file (
.child.yml
).
# .child.yml
build:
stage: build
script:
- echo "Do your build here"
- Add the following
.gitlab-ci.yml
file to the project:
normal job:
script: sleep 15
trigger a:
trigger:
include: ".child.yml"
strategy: depend
trigger b:
trigger:
include: ".child.yml"
strategy: depend
trigger c:
trigger:
include: ".child.yml"
strategy: depend
- Commit a change to the a file to kick off a new pipeline (we'll call this P1)
- Within 5-7 seconds of creating P1, commit another change to a file to create another pipeline (we'll call this P2)
- Navigate to the P1 view, and you'll see that child pipelines are canceled even though their jobs don't have
interruptible: true
.
Analysis
As explained above, in the "Any uninterruptible jobs have not started yet." scenario, we cancel the whole pipeline. And in the customer case, if any job of a child pipeline has not started yet, we cancel it.
Desired outcome
Users want to have an option to choose if the pipeline should be canceled or not if all jobs have not started yet.
Proposal
We'll introduce a new syntax to control how auto-cancel works;
workflow:
auto_cancel:
on_new_commit: conservative # (default), other options: "interruptible", "none"
options:
"all": when a new commit/pipeline is created, cancel all jobs.- "interruptible": when a new commit/pipeline is created, cancel only
interruptible:true
jobs. - "conservative": when a new commit/pipeline is created, behave the old/legacy way; check if there is at least one "started"
interruptible:false
job. If so, then don't cancel anything. If there is not, then cancel all jobs in the pipeline. - "none": when a new commit/pipeline is created, do not cancel any job in the pipeline.
Users will also have these options;
Either this (A):
default:
interruptible: true
workflow:
auto_cancel:
on_new_commit: interruptible
rules:
- if: $CI_COMMIT_REF_PROTECTED
auto_cancel:
on_new_commit: none
With this setting, we're keeping the interruptible
of jobs but we're changing the strategy of canceling based on CI_COMMIT_REF_PROTECTED
.
or this (B):
default:
interruptible: true
rules:
- if: $CI_COMMIT_REF_PROTECTED
interruptible: false
workflow:
auto_cancel:
on_new_commit: interruptible
With this setting, we're keeping the auto-canceling strategy but we're changing the default interruptible
for jobs based on CI_COMMIT_REF_PROTECTED
.
Implementation proposal
- Add
interruptible
to trigger jobs.
Currently, trigger jobs get auto-canceled if they are not started yet. However, when we implement and use on_new_commit: interruptible
, they will never get auto-canceled because trigger jobs can't have the interruptible
attribute. So, we also need to add the support of interruptible
to trigger jobs. However, we need to have a nice documentation of this to explain how this interruptible
affects trigger jobs (for example; interruptible
affects non-started trigger jobs).
- Add CI YAML config:
workflow:auto_cancel:on_new_commit
. - Add CI YAML config:
workflow:rules:auto_cancel:on_new_commit
.
Implementation table
- Add auto_cancel_on_new_commit to ci_pipeline_me... (!137520 - merged)
- Add new syntax of workflow:auto_cancel:on_new_c... (!137892 - merged)
- Add `interruptible` to trigger jobs (!138508 - merged)
- Add auto_cancel:on_new_commit to canceling redu... (!139358 - merged)
- workflow:rules:auto_cancel (#436467 - closed)