Cancel the pipeline immediately if any jobs fails
### Release Notes
You can now configure a pipeline to be cancelled immediately when a job fails. Please see documentation for additional details.Special thanks to @zillemarco for contributing to the feature.
### Problem to solve
We should allow the user to configure a pipeline in such a way that when one job fails it cancels all other already started jobs to reduce CI/CD minutes consumed when the pipeline would fail anyway.
## Proposal
### Acceptance Criteria
When
```yaml
workflow:
auto_cancel:
on_job_failure: all # or: none, default none
```
is configured then a single job failure will cause all jobs in the pipeline to be cancelled.
When this configuration is not included then the pipeline will behave as it does today (won't be cancelled).
Cascading to children is required but specify a cancellation reason is not required.
### Engineering
Looking at it again, I might even make the keyword more specific, to be `workflow:cancel_on_job_failure`.
My thinking is that `workflow:cancel_on_job_failure: all` would persist _somewhere_ attached to the Pipeline record, and then we'd check in the `BuildFinishedWorker`:
```diff
# frozen_string_literal: true
module Ci
class BuildFinishedWorker # rubocop:disable Scalability/IdempotentWorker
...
def process_build(build)
# We execute these in sync to reduce IO.
build.update_coverage
Ci::BuildReportResultService.new.execute(build)
build.execute_hooks
ChatNotificationWorker.perform_async(build.id) if build.pipeline.chat?
build.track_deployment_usage
build.track_verify_environment_usage
build.remove_token!
if build.failed? && !build.auto_retry_expected?
+ if build.pipeline.cancel_on_job_failure == :all # Something like this
+ ::Ci::CancelPipelineWorker.perform_async(build.pipeline_id, build.pipeline_id) # Add param for passing cascade_to_children: true?
+ end
+
::Ci::MergeRequests::AddTodoWhenBuildFailsWorker.perform_async(build.id)
end
```
```diff
module Ci
class Pipeline < Ci::ApplicationRecord
+ def cancel_on_job_failure
+ # Return the config value of the workflow:cancel_on_job_failure keyword.
+ # This can be a simple string ("all") for now, or become something more
+ # complicated later, e.g. { states: ['created', 'pending'] }
+ end
```
In the future, to extend this with more granular configuration, we'd replace the simple `pipeline.cancel_running` call with something more complex:
```yaml
cancel_on_job_failure:
state: [created, pending]
```
```diff
if build.failed? && !build.auto_retry_expected?
- build.pipeline.cancel_running if build.pipeline.cancel_on_job_failure == :all
+ cancel_all_immediately_cancellable_jobs if build.pipeline.cancel_on_job_failure.present?
::Ci::MergeRequests::AddTodoWhenBuildFailsWorker.perform_async(build.id)
end
end
+ def cancel_all_immediately_cancellable_jobs
+ # a bunch of query logic that passes configuration options into Ci::BuildCancelService, etc.
+ end
```
I'm not going to sketch out the whole future implementation here, but I'm just demonstrating that by inserting a check in the BuildFinishedWorker, we're working in a relatively flexible asynchronous place where we can make calls to queue other asynchronous querying and cancellation functionality.
### What is NOT included in the MVC
* A new failure type will not be reported, pipelines will report as canceled, jobs will report as canceled.
* No extra error reporting in the job log will be inserted.
* ~~Configurable through new Workflow syntax~~
```yaml
workflow:
auto_cancel:
on_job_failure: all
```
I scratched that last bullet point about workflow syntax, because a bunch of us got together and agreed that introducing a _very specific syntax_ directly under the `workflow` keyword is a relatively light-touch way to do exactly what's asked in this issue, while leaving us room to iterate and make improvements, more fine-tuned configuration, etc.
So for this (useful!) MVC, we will _only_ configure this at the `workflow` level, and have the only allowed value `all` apply to all jobs in the Pipeline.
#### Considerations
* It should be optional and off by default.
* Possibly a GitLab CI yaml level configuration.
* Deployment jobs should not be canceled for MVC.
#### How does this work with `interruptible`?
**Importantly, it does not.** `interruptible` configuration, while being named very generically, is a very specific functionality where a Pipeline may be cancelled by a _newer, different pipeline_ running on the same ref. That is the only application of it.
This change, specifically, is to enable Pipelines to cancel _themselves_ after a single job failure.
_If_ these two configurations are to intersect in the future, we'll have to decide how to do that. The naming of `interruptible` makes it somewhat difficult because there are so many different kinds of interruption that customers want. This is a future concern, and will not be addressed here.
### What does success look like, and how can we measure that?
* For GitLab pipelines we should see a decrease in time for failed pipelines by \>= 10%
* We'd expect to see XX pipelines on GitLab.com with this configuration added 30 days after GA
#### For the ~"internal customer"
* Create an ability to define a pipeline as `fail-fast`
* When a job fails, it should immediately cancel all running jobs in the pipeline and set the pipeline status to `failed`
#### For all users
* All other jobs of a pipeline are cancelled if one job fails.
### Links / references
- TravisCI is considering a similar feature - https://github.com/travis-ci/travis-ci/issues/2062
- Jenkins has a feature called `failFast` - https://jenkins.io/doc/book/pipeline/syntax/#parallel
_This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc._
_This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc._
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
*This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.*
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
issue