Ensure after_script is called for cancelled and timed out pipelines

Release notes

Problem to Solve

It's possible to want a after_script step for a gitlab-ci.yml, where it's important that it always run. However, the current behavior of after_script is that if a build is canceled or times out the cleanup stage is skipped (even though this does not appear to be the original intent: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/102#note_11619214).

Proposal

Backend, Step 1

We should add a new option for interruptible when configuring CI jobs. When interruptible: graceful is set after_script should be run on a canceled job.

Gitlab will communicate this with the runner through the requestJob endpoint. A new value will be added to step::when - graceful. The step for after_script will have when:graceful.

Runner, Step 1

When the runner sees a step with when: graceful it will run it even if the job is canceled. It will also upload artifacts.

If a job is in either created or pending, it should transition directly to canceled as no execution has happened yet.

Regarding execution context:

Concern: the configuration documentation specifies that after_script is run in a separate shell context. There's emphasis in the original issue that after_script execution needs to be tightly coupled with the job script specifically to have access to data (e.g. docker container IDs) from the script being cancelled. Is this going to be a problem? If so, is there any way to change that?

Response (@steveazz): I don't think is an issue, they will have all the environment variable that before_script & script has, the only thing that after_script will not have is if before_script/script create any environment variable of their own. They will be run on the same container just in a different context so they can't share information but this has always been like that from the start, users still has all the information needed to clean up their environment.

Gitlab Runner

Changes will be required in the runner, and gitlab-runner#4843 is open to track that.

Backend, Part 2

Part two will be mostly for handling traces on after_script for canceled jobs. The runner will continue sending them. Gitlab will introduce a new state: teardown which will allow traces to be sent and display them. This should be extended to Ci::Pipeline, where graceful cancellation sends the newly-implemented graceful cancellation to all of its builds. created and pending builds would be cancelled immediately, while running builds move immediately move on into their after_script.

Runner, Part 2

The runner should handle the new teardown state and send traces for it. gitlab-runner#4843 is also for tracking this.

Frontend

The main Cancel ❌ button on the Pipelines page should be a button for a graceful shutdown if interruptible: graceful.

Intended users

• Sasha (Software Developer)
• Devon (DevOps Engineer)

User experience goal

Proposal

Further details

Permissions and Security

Documentation

Availability & Testing

What does success look like, and how can we measure that?

What is the type of buyer?

Is this a cross-stage feature?

Links / references

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.