Allow delayed stage to run when previous task failed
Problem to solve
Improve the delayed stage functionality to allow it to run after a previous stage failed. This is useful when you want to have a cleanup stage in your dev pipeline and you also want to have a delay for troubleshooting purposes.
Target audience
Any developer building a temporary environment for his tests and requiring to tear it down afterward.
Further details
Candidate pipeline stages:
- Create infrastructure
- Test infrastructure
- Install software
- Configure software
- Test software
- Clean up dev environment
- If all previous stages were successful, run immediately
- If any stage failed, delay the clean up to allow for troubleshooting.
Currently, the only work around for a previously failed stage is the following:
stage: failed_wait
script:
- echo -e "You pipeline failed.\n\nBranch name = $CI_COMMIT_REF_NAME\nPipeline URL = $CI_PIPELINE_URL" | /usr/bin/heirloom-mailx -v -s "Pipeline Failed. Grace Period Started." $GITLAB_USER_EMAIL
- sleep 600
when: on_failure
allow_failure: true
- The email allows for immediate notification
- The sleep allows the dev to cancel the execution of the stage to troubleshoot
Problems:
- After cancelling the stage, to proceed with the clean up, we need to wait an additional 10 minutes
OR
- After cancelling the stage, we need to run the clean up stage manually from our laptop
Proposal
Currently the when
keyword works like this:
when: delayed
which is the equivalent of:
when:
- delayed
- on_success
Being able to list multiple values could allow parametrizing the stage:
# Runs only when a previous task failed
when:
- delayed
- on_failure
# Run whenever all previous stages passed or one of them failed
when:
- delayed
- always
What does success look like, and how can we measure that?
The following .gitlab-ci.yml files should both trigger the delayed task
stages:
- successful_stage
- delayed_stage
successful_task:
stage: successful_stage
script:
- exit 0
delayed_task:
stage: delayed_stage
script:
- exit 0
when:
- delayed
- always
start_in: 1 minute
stages:
- failed_stage
- delayed_stage
failed_task:
stage: failed_stage
script:
- exit 1
delayed_task:
stage: delayed_stage
script:
- exit 0
when:
- delayed
- always
start_in: 1 minute
Links / references
N/A