Backend: Allow `needs` to be used with `rules`
Release notes
Use needs
in rules to update a job's needs
for specific conditions. When a condition matches a rule, the job's needs
configuration is completely replaced with the needs
in the rule
Problem to solve
Original description
* Introduction of Rules and Needs features provide some incredibly powerful tools. * Implementing Needs becomes a task that requires duplicating a massive number of downstream jobs if they can't be dynamically set.A single downstream, with 2 optional presteps, quickly becomes 4 individual jobs that have to be mutually exclusive.
deploy release production:
extends:
- .deploy
- .release
- .production
deploy release production w/ prestep 1:
extends:
- .deploy
- .release
- .production
- .prestep1
deploy release production w/ prestep 2:
extends:
- .deploy
- .release
- .production
- .prestep2
deploy release production w/ prestep all:
extends:
- .deploy
- .release
- .production
- .prestepall
Allowing needs to be set with rules conditions can easily cut this down to one job that is much more predictable
deploy release production:
extends:
- .deploy
- .release
- .production
rules:
# rules allows for a list of individual rule objects to be evaluated in order,
# until one matches and dynamically provides attributes to the job.
- if: '$DEPLOY_PRESTEP1 && $DEPLOY_PRESTEP2'
needs: ["prestep1:release:production","prestep2:release:production"]
when: on_success
- if: '$DEPLOY_PRESTEP1'
needs: ["prestep1:release:production"]
when: on_success
- if: '$DEPLOY_PRESTEP2'
needs: ["prestep2:release:production"]
when: on_success
when: manual
Goal: We will allow rules to take needs which will be the exact set of keywords that we have under job's needs currently, see doc.
Intended users
- DevOps Engineer - flexible yet predictable templating and abstraction
Further details
- Reduce complexity and increase readability by reducing number of jobs.
What does success look like, and how can we measure that?
Definition of done: We can apply needs to rules with all the keywords that currently are available under job's needs.
Original proposal
### ProposalSupport the following syntax
test:
rules:
- if: '$SOMETING1'
needs: ["job1"]
- if: '$SOMETING2'
needs: ["job2"]
When '$SOMETHING1' is true, the test
job needs
will change to ["job1"]
When '$SOMETHING2' is true, the test
job needs
will change to ["job2"]
When neither 'SOMETHING1' or '
SOMETHING2' evaluates to true, the job wont have any needs defined
test:
needs: ['lint']
rules:
- if: '$SOMETHING1'
needs: ["job1"]
- if: '$SOMETHING2'
needs: ["job2"]
When '$SOMETHING1' is true, the test
job needs
will change to ["job1"]
When '$SOMETHING2' is true, the test
job needs
will change to ["job2"]
When neither 'SOMETHING1' or '
SOMETHING2' evaluates to true, the needs
will remain on ['lint']
if '$SOMETHING1' is true:
test:
needs: ['lint']
rules:
- if: '$SOMETHING1'
needs:
- job: "job1"
optional: true
- if: '$SOMETHING2'
needs: ["job2"]
it evaluates to:
test:
needs:
- job: ["job1"]
optional: true
Rules are evaluated when the pipeline is created, and evaluated in order until the first match
Technical approach: The job's needs logic will not be copied, but reused under rules needs so in the future any changes done to needs on the job's side, will be reflected under the rules needs and vice versa
Permissions and Security
N/A
MR addressing this issue. FF issue.
Updated proposal
Currently we have needs on jobs eg
build:
stage: build
script: echo
needs: [test]
In this issue, we are introducing needs on rules eg
lint_job:
stage: lint
script: 'echo lint_job'
needs: [test]
rules:
- if: $var == null
needs: [test_job]
or
lint_job:
stage: lint
script: 'echo lint_job'
needs: [test]
rules:
- if: $var == null
needs:
- job: test_job
artifacts: true
optional: false
If in the above lint_job
example, if the rule evaluates to true, the job's need gets overridden to test_job
. (In the later example, test_job attributes are also set: artifacts is set to true and option is set to false, if not specified, the default value is set)
Once specific condition in the rule is met, the needs from the rule override the needs on the job.
We will also allow the following keywords in the needs: artifacts, optional.
In this issue, we will not support the following options:
- https://docs.gitlab.com/ee/ci/yaml/#needsproject -> adding dependencies to download artifacts.
- https://docs.gitlab.com/ee/ci/yaml/#needspipeline -> provides mirroring pipeline status from an upstream pipeline to a bridge
- https://docs.gitlab.com/ee/ci/yaml/#needspipelinejob -> also downloading artifacts from a job in its parent pipeline or another child pipeline
Documentation
https://docs.gitlab.com/ee/ci/yaml/#permitted-attributes
Testing
- Create a job who's
needs
can be controlled using variable flags. - Toggle different variable flags.
Alternative testing: Create yml file with a pipeline and set rules to evaluate to true, push the code. Once the code is pushed, the yml will automatically run and the jobs needs to be evaluated with considerations on job needs.
What does success look like, and how can we measure that?
Definition of done: We can apply needs to rules with artifacts, optional keywords.