Flexible rules for Pipeline Jobs
Problem to solve
Today we use
only/except to define when to run jobs. The rigid structure of the existing
only/except logic coupled with the extension to simple vs. complex logic can cause it to be very hard to decide (as a human) when we expect a particular job to be run. Also, it also limits the ability to create what is a seemingly simple conjunction. (If this or that for instance). It also limits or makes it impossible to define when to run job flexibly. Consider the following example from our own configuration, and try to decipher when it is going to be run?
review-schedules: only: refs: - schedules@gitlab-org/gitlab-ce - schedules@gitlab-org/gitlab-ee kubernetes: active variables: - $REVIEW_APP_CLEANUP except: refs: - tags - /(^docs[\/-].*|.*-docs$)/
To simplify this, we will introduce a new syntax for defining rules, from which you can provide an array of matching criteria (first match wins), and to which a behavior is tied. The above example would instead look like:
review-schedules: rules: - branch: /(^docs[\/-].*|.*-docs$)/ when: never - branch: yes source: schedules project: gitlab-org/gitlab-ce if: $KUBECONFIG || $REVIEW_APP_CLEANUP
At the same time we will consider deprecating the
allow_failure syntax, with a possibility to remove in GitLab 13.0 (June 2020). It will not be possible to combine the old and new syntax in a single job: this will result in a job failure. Having some jobs that use
rules: and some jobs that use
only/except in the same
.gitlab-ci.yml is fine, though - this is supported to support composability of includes.
There are two parts to each rule - a criteria and an action (remember, first match "wins".) We'll start with the possible actions on match:
The default value of
when if not specified is
We have the following policy actions, most of them are related to what
when: offers today,
on_success: run when the previous stage succeeds, this replaces
on_failure: run when the previous stage fails, this replaces
always: run regardless of the previous stage, this replaces
never: do not run this CI job: this is how you implement skipping a job,
manual: run on manual request, this replaces
delayed: run job delayed, use with conjuction of
start_in:3 minutes (https://docs.gitlab.com/ee/ci/yaml/#whendelayed)
rules: - changes: - *.rb when: delayed start_in: 10 minutes
In order to know when to perform the action specified, we use matchers to decide. Each rule can have one or multiple matchers defined, and if multiple matchers are defined they are AND between each other (i.e., every match needs to be true.) Again, remember that the first match in a list wins.
Runs the if condition, re-uses variables: syntax.
rules: - if: $CI_COMMIT_TITLE =~ /run me/ || $CI_COMMIT_TITLE =~ /run2 me/
Runs on changes replicate existing changes: (https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges). It can accept either a string or an array of strings:
rules: changes: *.rb rules: - changes: - *.rb - Gemfile - Gemfile.lock
Other Matchers and Modifiers (out of scope)
There are also several other matchers that we could implement. All of them are implementable via
if: in combination with available predefined environment variables, but it may be syntactically nicer and easier to read if we offered unique keywords that handled it for you.
$CI_COMMIT_REF_NAME == "master" && $CI_COMMIT_TAG == "",
Because all of these can be implemented via
if:, they are not included in the MVC. There is an issue https://gitlab.com/gitlab-org/gitlab-ce/issues/63827 where we can discuss potentially adding these or other common matchers as syntactic sugar.
Other Actions (out of scope)
allow_failure:is the only other action proposed at this time, which when set would allow a job to fail. This will be implemented in the follow up issue https://gitlab.com/gitlab-org/gitlab-ce/issues/64796. The workaround for now is to use the existing allow_failure keyword on the job level (i.e., outside the
rules: - changes: *.doc allow_failure: true