Ensure after_script is called for cancelled and timed out pipelines
Release notes
Problem to Solve
It's possible to want a after_script step for a gitlab-ci.yml, where it's important that it always run. However, the current behavior of after_script is that if a build is canceled or times out the cleanup stage is skipped (even though this does not appear to be the original intent: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/102#note_11619214).
Proposal
Backend, Step 1
We should add a new option for interruptible when configuring CI jobs. When interruptible: graceful is set after_script should be run on a canceled job.
Gitlab will communicate this with the runner through the requestJob endpoint. A new value will be added to step::when - graceful. The step for after_script will have when:graceful.
Runner, Step 1
When the runner sees a step with when: graceful it will run it even if the job is canceled. It will also upload artifacts.
If a job is in either created or pending, it should transition directly to canceled as no execution has happened yet.
Regarding execution context:
Concern: the configuration documentation specifies that
after_scriptis run in a separate shell context. There's emphasis in the original issue thatafter_scriptexecution needs to be tightly coupled with the job script specifically to have access to data (e.g. docker container IDs) from the script being cancelled. Is this going to be a problem? If so, is there any way to change that?
Response (@steveazz): I don't think is an issue, they will have all the environment variable that
before_script&scripthas, the only thing thatafter_scriptwill not have is ifbefore_script/scriptcreate any environment variable of their own. They will be run on the same container just in a different context so they can't share information but this has always been like that from the start, users still has all the information needed to clean up their environment.
Gitlab Runner
Changes will be required in the runner, and gitlab-runner#4843 is open to track that.
Backend, Part 2
Part two will be mostly for handling traces on after_script for canceled jobs. The runner will continue sending them. Gitlab will introduce a new state: teardown which will allow traces to be sent and display them.
This should be extended to Ci::Pipeline, where graceful cancellation sends the newly-implemented graceful cancellation to all of its builds. created and pending builds would be cancelled immediately, while running builds move immediately move on into their after_script.
Runner, Part 2
The runner should handle the new teardown state and send traces for it. gitlab-runner#4843 is also for tracking this.
Frontend
The main Cancel interruptible: graceful.