Support CI `paths`, `project`, and `ref` subkeys in `rules:exists`
What does this MR do and why?
This MR augments rules:exists
so that it supports the subkeys: paths
, project
, and ref
. This enables the user to specify the exact project/ref in which to find the specified files, which circumvents the problem that occurs due to the different default contexts of job:rules:exists
and include:rules:exists
.
The implementation is as follows:
-
rules:exists
is supported both in a job as well as in aninclude
. -
rules:exists:paths
is the same as usingrules:exists
without any subkeys. -
exists:ref
can only be specified ifexists:project
is specified. - If
exists:project
is specified withoutexists:ref
, the default ref is the project'sHEAD
. - The current user must have
:read_code
ability on the specifiedexists:project
. - If
exists:project
is not specified, the default contexts are as they are currently (unchanged). Specifically:- For
job:rules:exists
, the default context is the project/ref in which the pipeline is running. - For
include:rules:exists
, the default context is the project/ref of the file in which theinclude
is defined.
- For
Feature flag:
These changes are made behind FF ci_support_rules_exists_paths_and_project
. Roll-out issue: #453983 (closed).
Follow-up issues:
- To clean up the code: Backend: Refactor Rules::Rule::Clause classes t... (#454384 - closed)
- To improve
rules:exists:project
performance by making batched requests: Backend: Refactor Rule::Clause::Exists to impro... (#454595 - closed)
Example usage:
include:
- local: hello.yml
rules:
- exists:
paths: [abc.md, cd.md]
project: root/my-project
ref: main
job:
script: exit 0
rules:
- exists:
paths: [xyz.yml]
project: root/my-project
ref: other_branch
Resolves #386040 (closed).
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
Preparation:
- Create a new public project and private project named
Public Project
andPrivate Project
, respectively. In the examples below, I have created the projects under a public group namedGroup E
(group-e
). Please adjust the group path when you're testing as necessary. - In the
Public Project
root, onmain
, create the following files:
public-my_file.txt
:
content can be anything
job1.yml
:
job1:
script: echo
job2.yml
:
job2:
script: echo
job3.yml
:
job3:
script: echo
- In the
Private Project
root:
- On
main
, create the following file:
private-my_file.txt
:
content can be anything
- Create a new branch off of
main
namedother_branch
. On this new branch, renameprivate-my_file.txt
toprivate-my_file_on_other_branch.txt
.
- Go to Admin Area and create a new regular user named
Test User
. Assign this user as anOwner
toPublic Project
. We will impersonate this user in a later step.
rules:exists
with FF off
Test - With the feature flag OFF, let's first test that everything works normally as before.
- In
Public Project
's Pipeline editor, update the content with:
include:
- local: job3.yml
rules:
- exists:
- public-my_file.txt
job4:
script: echo
rules:
- exists:
- public-my_file.txt
There should be no validation errors in the Pipeline editor.
Commit and run the pipeline. Observe that both job3
and job4
are included in the pipeline as expected.
- Now in the Pipeline editor, add the
path
subkey toexists
; observe that there is a validation error as expected.
rules:exists:paths
Test - In the Rails console, enable the feature flag:
Feature.enable(:ci_support_rules_exists_paths_and_project)
. - In
Public Project
's Pipeline editor, test the following:
include:
- local: job2.yml
rules:
- exists:
- public-my_file.txt
- local: job3.yml
rules:
- exists:
paths:
- public-my_file.txt
job4:
script: echo
rules:
- exists:
- public-my_file.txt
job5:
script: echo
rules:
- exists:
paths:
- public-my_file.txt
Commit and run the pipeline. Observe that the inclusion of all four jobs (job2-5
) indicate that exists:paths
works exactly the same as exists
without subkeys.
rules:exists:project
Test - Ensure the feature flag is ON.
- In
Public Project
's Pipeline editor, test the following:
include:
- local: job2.yml
rules:
- exists: # Checks in group-e/public-project on `main`
- private-my_file.txt
- local: job3.yml
rules:
- exists:
paths:
- private-my_file.txt
project: group-e/private-project
ref: main
job4:
script: echo
rules:
- exists: # Checks in group-e/public-project on `main`
- private-my_file.txt
job5:
script: echo
rules:
- exists:
paths:
- private-my_file.txt
project: group-e/private-project
ref: main
Commit and run the pipeline. The exist rules for job2
and job4
attempt to check for private-my_file.txt
in Public Project
, which does not exist. So we only expect job3
and job5
in the pipeline.
- Let's test a more complicated example involving nested includes. In
Private Project
onother_branch
, create the following config file:
config.yml
:
include:
- project: group-e/public-project
file: job1.yml
rules:
- exists: # Checks in the current file's context: private-project on `other_branch`. Neither of these files exist there, so this job is not added.
- public-my_file.txt
- private-my_file.txt
- project: group-e/public-project
file: job2.yml
rules:
- exists:
paths:
- private-my_file.txt
project: group-e/private-project
- project: group-e/public-project
file: job3.yml
rules:
- exists:
paths:
- public-my_file.txt
project: group-e/public-project
ref: main
job4:
script: echo
rules:
- exists: # Checks in the pipeline's context: group-e/public-project on `main`. This file does exist there, so this file is added.
paths:
- public-my_file.txt
job5:
script: echo
rules:
- exists:
paths:
- private-my_file.txt
project: group-e/private-project
job6:
script: echo
rules:
- exists:
paths:
- private-my_file.txt
project: group-e/private-project
ref: main
- Back in
Public Project
's Pipeline editor, test the following:
include:
- project: group-e/private-project
ref: other_branch
file: config.yml
Commit and run the pipeline. We expect all jobs to be included except for job1
(job2-6
).
rules:exists:project
when the user does not have access
Test - Ensure the feature flag is ON.
- Go to the Admin Area and find the
Test User
you created earlier. Click into see its details and click the "impersonate" button at the top right. - Navigate to
Public Project
's Pipeline editor and test the following:
include:
- local: job1.yml
rules:
- exists:
paths:
- private-my_file.txt
project: group-e/private-project
Observe there is a validation error indicating access is denied.
- Now try it with a job. Update the Pipeline editor's content with the following:
job:
script: echo
rules:
- exists:
paths:
- private-my_file.txt
project: group-e/private-project
ref: main
control:
script: echo
Commit the and run the pipeline. Observe that there is a pipeline error indicating access is denied.
- Make sure to stop impersonating the user after this test.
.gitlab-ci.yml
Test a Compliance pipeline in a project that does not have a When utilizing compliance pipelines, users often want to run both the compliance Yaml as well as the pipeline configuration of the project running the pipeline. They are currently recommended to support this with include:project
as shown in this example configuration:
include: # Execute individual project's configuration (if project contains .gitlab-ci.yml)
- project: '$CI_PROJECT_PATH'
file: '$CI_CONFIG_PATH'
ref: '$CI_COMMIT_SHA'
The problem with the above configuration is that it breaks when the included project does not have a pipeline configuration file (typically .gitlab-ci.yml
).
Prior to this MR, the users would try to use rules:exists
like so:
include:
- project: '$CI_PROJECT_PATH'
file: '$CI_CONFIG_PATH'
ref: '$CI_COMMIT_SHA'
rules:
- exists:
- '$CI_CONFIG_PATH'
However, it does not work because the default search context of include:rules:exists
is the project in which the include
is defined. So in this case, it's attempting to look for the file in the project hosting the Compliance Yaml instead of the project running the pipeline.
This MR now solves this problem by allowing the user to specify exactly the context to search in. And because it supports expanding variables in the subkeys, we can now do this in the compliance Yaml:
include:
- project: '$CI_PROJECT_PATH'
file: '$CI_CONFIG_PATH'
ref: '$CI_COMMIT_SHA'
rules:
- exists:
paths:
- '$CI_CONFIG_PATH'
project: '$CI_PROJECT_PATH'
ref: '$CI_COMMIT_SHA'
Testing steps:
- Ensure the FF is ON.
- Create a new blank project named
My Project
. In this project:
- Go to Settings > CI/CD > Auto DevOps and ensure "Default to Auto DevOps pipeline" is NOT enabled.
- Create a new blank project named
Compliance Project
. In this project:
- Create the following files:
compliance.yml
:
# Old configuration that breaks the pipeline on projects without a configuration file
include:
- project: '$CI_PROJECT_PATH'
file: '$CI_CONFIG_PATH'
ref: '$CI_COMMIT_SHA'
rules:
- exists: # Checks in the current file's context: compliance-project on `main`. It sees that `.gitlab-ci.yml` exists and attempts to include the project's config file, but it's not there. So it breaks the pipeline.
- '$CI_CONFIG_PATH'
compliance-job:
script: echo
.gitlab-ci.yml
:
# We just need this file to exist in the Compliance Project
- In the group in which you created the earlier projects:
- Go to Settings > General.
- Expand the Compliance frameworks section.
- In Compliance pipeline configuration (optional), add a new framework with name
My Compliance
with the path to the compliance Yaml: e.g.compliance.yml@group-e/compliance-project
- In
My Project
, now apply the compliance framework label.
- Go to Settings > Compliance framework. Select and apply
My Compliance
.
- In
My Project
, go to Pipelines and run a new pipeline. Observe that you get the following error:
- Now update
compliance.yml
with the following:
# New configuration that does not break the pipeline
include:
- project: '$CI_PROJECT_PATH'
file: '$CI_CONFIG_PATH'
ref: '$CI_COMMIT_SHA'
rules:
- exists: # Checks in the pipeline project's context: my-project on `main`. It sees that it does not have a config file, so it does not attempt to include it.
paths:
- '$CI_CONFIG_PATH'
project: '$CI_PROJECT_PATH'
ref: '$CI_COMMIT_SHA'
compliance-job:
script: echo
- In
My Project
, go to Pipelines and run a new pipeline. Observe that now the pipeline runs successfully, with only thecompliance-job
in the pipeline.
Related to #386040 (closed)
Merge request reports
Activity
changed milestone to %16.11
assigned to @lma-git
added 1 commit
- 6834465a - Support rules:exists subkeys `paths`, `project`, and `ref`
- A deleted user
added feature flagexists label
- Resolved by Leaminn Ma
2 Warnings This merge request is quite big (1192 lines changed), please consider splitting it into multiple merge requests. e8d8dabb: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. 2 Messages CHANGELOG missing: If this merge request needs a changelog entry, add the
Changelog
trailer to the commit message you want to add to the changelog.If this merge request doesn't need a CHANGELOG entry, feel free to ignore this message.
This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge. Documentation review
The following files require a review from a technical writer:
-
doc/ci/yaml/includes.md
(Link to current live version) -
doc/ci/yaml/index.md
(Link to current live version) -
doc/user/group/compliance_pipelines.md
(Link to current live version)
The review does not need to block merging this merge request. See the:
-
Metadata for the
*.md
files that you've changed. The first few lines of each*.md
file identify the stage and group most closely associated with your docs change. - The Technical Writer assigned for that stage and group.
- Documentation workflows for information on when to assign a merge request for review.
Reviewer roulette
Category Reviewer Maintainer backend @ibaum
(UTC-4, 3 hours ahead of author)
@pshutsin
(UTC+2, 9 hours ahead of author)
frontend @lorenzvanherwaarden
(UTC+2, 9 hours ahead of author)
@svedova
(UTC+2, 9 hours ahead of author)
~"Verify" Reviewer review is optional for ~"Verify" @mfluharty
(UTC-6, 1 hour ahead of author)
~"Verify" Reviewer review is optional for ~"Verify" @ayufan
(UTC+2, 9 hours ahead of author)
Please check reviewer's status!
Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Danger-
- Resolved by Leaminn Ma
- Resolved by Leaminn Ma
added 1 commit
- 1359234b - Support rules:exists subkeys `paths`, `project`, and `ref`
- Resolved by Leaminn Ma
Hi @furkanayhan!
Could you please give this MR a pre-review?
It's not fully complete with tests and it needs a few more adjustments, but I wanted to first get your initial thoughts on the approach before moving forward.This MR solves the problem with
include:rules:exists
using the proposal that we discussed in #386040 (comment 1582069978). I also supportedexists:ref
in this MR because I realized that it doesn't really make sense to specifyproject
withoutref
when we're looking for the file.
requested review from @furkanayhan
added 105 commits
-
1359234b...b3bdc16a - 104 commits from branch
master
- 7d9aa74c - Support rules:exists subkeys `paths`, `project`, and `ref`
-
1359234b...b3bdc16a - 104 commits from branch
added 1 commit
- 4d1da943 - Add docs for exists:paths and exists:project
- Resolved by Leaminn Ma
- Resolved by Leaminn Ma
- Resolved by Leaminn Ma
- Resolved by Leaminn Ma