Skip to content

Break down `only/except:refs:` keywords

Problem

only/except: refs: is a handy keyword to leverage job execution, however, often it becomes problematic when in complicated gitlab-ci.yml case. In fact, we tried to enable pipelines for merge requests in our development workflow but concluded that it's not maintainable. To illustrate the problems:

Case 1) I want to run a job for master branch or merge request on all gitlab canonical projects.

only:
  refs:
  - master@gitlab-org/gitlab-ce
  - master@gitlab-org/gitlab-ee
  - master@gitlab/gitlabhq
  - master@gitlab/gitlab-ee
  - merge_requests@gitlab-org/gitlab-ce
  - merge_requests@gitlab-org/gitlab-ee
  - merge_requests@gitlab/gitlabhq
  - merge_requests@gitlab/gitlab-ee

=> Repeating project paths. They have to repeat for the number of project paths, which increases the risk of typos.

Case 2) I want to run a job as Pipelines for merge requests which has a branch name started from docs-.

only:
  refs:
  - /docs-/
  - merge_requests

Of cource, the above syntax doesn't work. But we have a workaround

only:
  refs:
  - merge_requests
  variables:
  - $CI_COMMIT_REF_NAME =~ /docs-/

=> We have to use variables. Is it good UX experience?. Also users cannot use variables for the other conbination because we already used it for $CI_COMMIT_REF_NAME :(

Case 3) I want to run a job as Pipelines for merge requests which has a branch name started from docs- in gitlab canocical projects.

only:
  refs:
  - merge_requests@gitlab-org/gitlab-ce
  - merge_requests@gitlab-org/gitlab-ee
  - merge_requests@gitlab/gitlabhq
  - merge_requests@gitlab/gitlab-ee
  variables:
  - $CI_COMMIT_REF_NAME =~ /docs-/

=> Ah, there you go. You cannot use $CI_PROJECT_PATH :( so you have to repeat merge_requests@ for 4 times!

Case 4) I want to run a job on all protected branches/tags in gitlab canonical projects.

only:
  refs:
  - master@gitlab-org/gitlab-ce
  - master@gitlab-org/gitlab-ee
  - master@gitlab/gitlab-hq
  - master@gitlab/gitlab-ee
  - /^security-/@gitlab-org/gitlab-ce
  - /^security-/@gitlab-org/gitlab-ee
  - /^security-/@gitlab/gitlab-hq
  - /^security-/@gitlab/gitlab-ee
  - /^v\d+.\d+/@gitlab-org/gitlab-ce
  - /^v\d+.\d+/@gitlab-org/gitlab-ee
  - /^v\d+.\d+/@gitlab/gitlab-hq
  - /^v\d+.\d+/@gitlab/gitlab-ee
  - (... and if we added a new protected ref, then we **manually** add here)

=> Who can maintain this?

NOTE: Those examples are extracted from https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24880, which is an actual attempt to enable piplines for merge requests in our development workflow

Proposal

Break down only:refs keywords into only:branches, only:protected_branches, only:tags, only:merge_requests, only:events, only:project_paths. It works as following:

  • only:branches: [value, ...] ... If a job is a run on a branch which contains the specified pattern(s), then execute the job.
  • only:protected_branches: [value, ...] ... It rejects if the branch is not protected. Other than that, it's same behavior with only:branches.
  • only:tags: [value, ...] ... If a job is a run on a tag which contains the specified pattern(s), then execute the job.
  • only:protected_tags: [value, ...] ... It rejects if the tag is not protected. Other than that, it's same behavior with only:tags.
  • only:merge_requests: [value, ...] ... If a job is a run in merge request context and the source branch name matches the specified pattern, then execute the job.
  • only:events: [value, ...] ... If a job is triggered by one of the specified events, then execute the job. (Effectively, addresses https://gitlab.com/gitlab-org/gitlab-ce/issues/57075)
  • only:project_paths: [value, ...] ... If a job is triggered in one of the specified project paths, then execute the job.

and we introduce only/except: exactly_one_of special keyword, which effectively makes only:* keywords OR behavior instead of AND. (Inspried by Grape's exactly_one_of)

How it's going to work?

Case 1) I want to run a job for master branch or merge request on all gitlab canonical projects.

First, let's make YAML anchor (a.k.a job template) for project paths.

.only_gitlab_canonical_project_paths: &only_gitlab_canonical_project_paths
only:
  project_paths:
  - gitlab-org/gitlab-ce
  - gitlab-org/gitlab-ee
  - gitlab/gitlabhq
  - gitlab/gitlab-ee

then, we can

only:
  branches:
  - master
  merge_request:
  - /.+/
  exactly_one_of: [branches, events]
<< :only_gitlab_canonical_project_paths  # We can reuse the YAML anchor (a.k.a. job template)

Case 2) I want to run a job as Pipelines for merge requests which has a branch name started from docs-.

only:
  merge_requests:
  - /docs-/

Case 3) I want to run a job as Pipelines for merge requests which has a branch name started from docs- in gitlab canocical projects.

only:
  merge_requests:
  - /docs-/
<<: *only_gitlab_canonical_project_paths

Case 4) I want to run a job on all protected branches/tags in gitlab canonical projects.

only:
  protected_branches: /.+/
  protected_tags: /.+/
<<: *only_gitlab_canonical_project_paths

Bonus: Reference for only/except: refs:

  • refs: [tags] ... If the job is a run on a git tag, then execute the job.
  • refs: [branches] ... If the job is a run on a branch, then execute the job.
  • refs: [master] ... If the job is a run on a branch, then execute the job.
  • refs: [/^security-/] ... If the job is run on a branch or tag and the ref name fulfilled the RegExp condition, then execute the job.
  • refs: [schedules] ... If the job is triggered by a pipeline schedule, then execute the job.
  • refs: [branches@gitlab-org/gitlab-ce] ... If the job is run on a branch and created in the specified project path, then execute the job.