Break down `only/except:refs:` keywords
Problem
only/except: refs:
is a handy keyword to leverage job execution, however, often it becomes problematic when in complicated gitlab-ci.yml case. In fact, we tried to enable pipelines for merge requests in our development workflow but concluded that it's not maintainable. To illustrate the problems:
Case 1) I want to run a job for master branch or merge request on all gitlab canonical projects.
only:
refs:
- master@gitlab-org/gitlab-ce
- master@gitlab-org/gitlab-ee
- master@gitlab/gitlabhq
- master@gitlab/gitlab-ee
- merge_requests@gitlab-org/gitlab-ce
- merge_requests@gitlab-org/gitlab-ee
- merge_requests@gitlab/gitlabhq
- merge_requests@gitlab/gitlab-ee
=> Repeating project paths. They have to repeat for the number of project paths, which increases the risk of typos.
Pipelines for merge requests which has a branch name started from docs-
.
Case 2) I want to run a job as only:
refs:
- /docs-/
- merge_requests
Of cource, the above syntax doesn't work. But we have a workaround
only:
refs:
- merge_requests
variables:
- $CI_COMMIT_REF_NAME =~ /docs-/
=> We have to use variables
. Is it good UX experience?. Also users cannot use variables
for the other conbination because we already used it for $CI_COMMIT_REF_NAME :(
Pipelines for merge requests which has a branch name started from docs-
in gitlab canocical projects.
Case 3) I want to run a job as only:
refs:
- merge_requests@gitlab-org/gitlab-ce
- merge_requests@gitlab-org/gitlab-ee
- merge_requests@gitlab/gitlabhq
- merge_requests@gitlab/gitlab-ee
variables:
- $CI_COMMIT_REF_NAME =~ /docs-/
=> Ah, there you go. You cannot use $CI_PROJECT_PATH
:( so you have to repeat merge_requests@
for 4 times!
Case 4) I want to run a job on all protected branches/tags in gitlab canonical projects.
only:
refs:
- master@gitlab-org/gitlab-ce
- master@gitlab-org/gitlab-ee
- master@gitlab/gitlab-hq
- master@gitlab/gitlab-ee
- /^security-/@gitlab-org/gitlab-ce
- /^security-/@gitlab-org/gitlab-ee
- /^security-/@gitlab/gitlab-hq
- /^security-/@gitlab/gitlab-ee
- /^v\d+.\d+/@gitlab-org/gitlab-ce
- /^v\d+.\d+/@gitlab-org/gitlab-ee
- /^v\d+.\d+/@gitlab/gitlab-hq
- /^v\d+.\d+/@gitlab/gitlab-ee
- (... and if we added a new protected ref, then we **manually** add here)
=> Who can maintain this?
NOTE: Those examples are extracted from https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24880, which is an actual attempt to enable piplines for merge requests in our development workflow
Proposal
Break down only:refs
keywords into only:branches
, only:protected_branches
, only:tags
, only:merge_requests
, only:events
, only:project_paths
. It works as following:
-
only:branches: [value, ...]
... If a job is a run on a branch which contains the specified pattern(s), then execute the job. -
only:protected_branches: [value, ...]
... It rejects if the branch is not protected. Other than that, it's same behavior withonly:branches
. -
only:tags: [value, ...]
... If a job is a run on a tag which contains the specified pattern(s), then execute the job. -
only:protected_tags: [value, ...]
... It rejects if the tag is not protected. Other than that, it's same behavior withonly:tags
. -
only:merge_requests: [value, ...]
... If a job is a run in merge request context and the source branch name matches the specified pattern, then execute the job. -
only:events: [value, ...]
... If a job is triggered by one of the specified events, then execute the job. (Effectively, addresses https://gitlab.com/gitlab-org/gitlab-ce/issues/57075) -
only:project_paths: [value, ...]
... If a job is triggered in one of the specified project paths, then execute the job.
and we introduce only/except: exactly_one_of
special keyword, which effectively makes only:*
keywords OR
behavior instead of AND
. (Inspried by Grape's exactly_one_of)
How it's going to work?
Case 1) I want to run a job for master branch or merge request on all gitlab canonical projects.
First, let's make YAML anchor (a.k.a job template) for project paths.
.only_gitlab_canonical_project_paths: &only_gitlab_canonical_project_paths
only:
project_paths:
- gitlab-org/gitlab-ce
- gitlab-org/gitlab-ee
- gitlab/gitlabhq
- gitlab/gitlab-ee
then, we can
only:
branches:
- master
merge_request:
- /.+/
exactly_one_of: [branches, events]
<< :only_gitlab_canonical_project_paths # We can reuse the YAML anchor (a.k.a. job template)
Pipelines for merge requests which has a branch name started from docs-
.
Case 2) I want to run a job as only:
merge_requests:
- /docs-/
Pipelines for merge requests which has a branch name started from docs-
in gitlab canocical projects.
Case 3) I want to run a job as only:
merge_requests:
- /docs-/
<<: *only_gitlab_canonical_project_paths
Case 4) I want to run a job on all protected branches/tags in gitlab canonical projects.
only:
protected_branches: /.+/
protected_tags: /.+/
<<: *only_gitlab_canonical_project_paths
only/except: refs:
Bonus: Reference for -
refs: [tags]
... If the job is a run on a git tag, then execute the job. -
refs: [branches]
... If the job is a run on a branch, then execute the job. -
refs: [master]
... If the job is a run on a branch, then execute the job. -
refs: [/^security-/]
... If the job is run on a branch or tag and the ref name fulfilled the RegExp condition, then execute the job. -
refs: [schedules]
... If the job is triggered by a pipeline schedule, then execute the job. -
refs: [branches@gitlab-org/gitlab-ce]
... If the job is run on a branch and created in the specified project path, then execute the job.