Support CI `paths`, `project`, and `ref` subkeys in `rules:exists`

What does this MR do and why?

This MR augments rules:exists so that it supports the subkeys: paths, project, and ref. This enables the user to specify the exact project/ref in which to find the specified files, which circumvents the problem that occurs due to the different default contexts of job:rules:exists and include:rules:exists.

The implementation is as follows:

• rules:exists is supported both in a job as well as in an include.
• rules:exists:paths is the same as using rules:exists without any subkeys.
• exists:ref can only be specified if exists:project is specified.
• If exists:project is specified without exists:ref, the default ref is the project's HEAD.
• The current user must have :read_code ability on the specified exists:project.
• If exists:project is not specified, the default contexts are as they are currently (unchanged). Specifically:
  • For job:rules:exists, the default context is the project/ref in which the pipeline is running.
  • For include:rules:exists, the default context is the project/ref of the file in which the include is defined.

Feature flag:

These changes are made behind FF ci_support_rules_exists_paths_and_project. Roll-out issue: #453983 (closed).

Follow-up issues:

• To clean up the code: Backend: Refactor Rules::Rule::Clause classes t... (#454384 - closed)
• To improve rules:exists:project performance by making batched requests: Backend: Refactor Rule::Clause::Exists to impro... (#454595 - closed)

Example usage:

include:
  - local: hello.yml
    rules:
      - exists:
          paths: [abc.md, cd.md]
          project: root/my-project
          ref: main

job:
  script: exit 0
  rules:
    - exists:
        paths: [xyz.yml]
        project: root/my-project
        ref: other_branch

Resolves #386040 (closed).

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

Preparation:

1. Create a new public project and private project named Public Project and Private Project, respectively. In the examples below, I have created the projects under a public group named Group E (group-e). Please adjust the group path when you're testing as necessary.
2. In the Public Project root, on main, create the following files:

public-my_file.txt:

content can be anything

job1.yml:

job1:
  script: echo

job2.yml:

job2:
  script: echo

job3.yml:

job3:
  script: echo

3. In the Private Project root:

• On main, create the following file:

private-my_file.txt:

content can be anything

• Create a new branch off of main named other_branch. On this new branch, rename private-my_file.txt to private-my_file_on_other_branch.txt.

4. Go to Admin Area and create a new regular user named Test User. Assign this user as an Owner to Public Project. We will impersonate this user in a later step.

Test rules:exists with FF off

1. With the feature flag OFF, let's first test that everything works normally as before.
2. In Public Project's Pipeline editor, update the content with:

include:
  - local: job3.yml
    rules:
      - exists:
          - public-my_file.txt

job4:
  script: echo
  rules:
    - exists:
        - public-my_file.txt

There should be no validation errors in the Pipeline editor.

Screenshot

Commit and run the pipeline. Observe that both job3 and job4 are included in the pipeline as expected.

3. Now in the Pipeline editor, add the path subkey to exists; observe that there is a validation error as expected.

Screenshot

Test rules:exists:paths

1. In the Rails console, enable the feature flag: Feature.enable(:ci_support_rules_exists_paths_and_project).
2. In Public Project's Pipeline editor, test the following:

include:
  - local: job2.yml
    rules:
      - exists:
          - public-my_file.txt
  - local: job3.yml
    rules:
      - exists:
          paths:
            - public-my_file.txt

job4:
  script: echo
  rules:
    - exists:
        - public-my_file.txt

job5:
  script: echo
  rules:
    - exists:
        paths:
          - public-my_file.txt

Commit and run the pipeline. Observe that the inclusion of all four jobs (job2-5) indicate that exists:paths works exactly the same as exists without subkeys.

Screenshot

Test rules:exists:project

1. Ensure the feature flag is ON.
2. In Public Project's Pipeline editor, test the following:

include:
  - local: job2.yml
    rules:
      - exists: # Checks in group-e/public-project on `main`
          - private-my_file.txt
  - local: job3.yml
    rules:
      - exists:
          paths:
            - private-my_file.txt
          project: group-e/private-project
          ref: main

job4:
  script: echo
  rules:
    - exists: # Checks in group-e/public-project on `main`
        - private-my_file.txt

job5:
  script: echo
  rules:
    - exists:
        paths:
          - private-my_file.txt
        project: group-e/private-project
        ref: main

Commit and run the pipeline. The exist rules for job2 and job4 attempt to check for private-my_file.txt in Public Project, which does not exist. So we only expect job3 and job5 in the pipeline.

3. Let's test a more complicated example involving nested includes. In Private Project on other_branch, create the following config file:

config.yml:

include:
  - project: group-e/public-project
    file: job1.yml
    rules:
      - exists: # Checks in the current file's context: private-project on `other_branch`. Neither of these files exist there, so this job is not added.
          - public-my_file.txt
          - private-my_file.txt
  - project: group-e/public-project
    file: job2.yml
    rules:
      - exists:
          paths:
            - private-my_file.txt
          project: group-e/private-project
  - project: group-e/public-project
    file: job3.yml
    rules:
      - exists:
          paths:
            - public-my_file.txt
          project: group-e/public-project
          ref: main

job4:
  script: echo
  rules:
    - exists: # Checks in the pipeline's context: group-e/public-project on `main`. This file does exist there, so this file is added.
        paths:
          - public-my_file.txt

job5:
  script: echo
  rules:
    - exists:
        paths:
          - private-my_file.txt
        project: group-e/private-project

job6:
  script: echo
  rules:
    - exists:
        paths:
          - private-my_file.txt
        project: group-e/private-project
        ref: main

4. Back in Public Project's Pipeline editor, test the following:

include:
  - project: group-e/private-project
    ref: other_branch
    file: config.yml

Commit and run the pipeline. We expect all jobs to be included except for job1 (job2-6).

Test rules:exists:project when the user does not have access

1. Ensure the feature flag is ON.
2. Go to the Admin Area and find the Test User you created earlier. Click into see its details and click the "impersonate" button at the top right.
3. Navigate to Public Project's Pipeline editor and test the following:

include:
  - local: job1.yml
    rules:
      - exists:
          paths:
            - private-my_file.txt
          project: group-e/private-project

Observe there is a validation error indicating access is denied.

Screenshot

4. Now try it with a job. Update the Pipeline editor's content with the following:

job:
  script: echo
  rules:
    - exists:
        paths:
          - private-my_file.txt
        project: group-e/private-project
        ref: main

control:
  script: echo

Commit the and run the pipeline. Observe that there is a pipeline error indicating access is denied.

Screenshot

5. Make sure to stop impersonating the user after this test.

Test a Compliance pipeline in a project that does not have a .gitlab-ci.yml

When utilizing compliance pipelines, users often want to run both the compliance Yaml as well as the pipeline configuration of the project running the pipeline. They are currently recommended to support this with include:project as shown in this example configuration:

include:  # Execute individual project's configuration (if project contains .gitlab-ci.yml)
  - project: '$CI_PROJECT_PATH'
    file: '$CI_CONFIG_PATH'
    ref: '$CI_COMMIT_SHA'

The problem with the above configuration is that it breaks when the included project does not have a pipeline configuration file (typically .gitlab-ci.yml).

Prior to this MR, the users would try to use rules:exists like so:

include:
  - project: '$CI_PROJECT_PATH'
    file: '$CI_CONFIG_PATH'
    ref: '$CI_COMMIT_SHA'
    rules:
      - exists:
          - '$CI_CONFIG_PATH'

However, it does not work because the default search context of include:rules:exists is the project in which the include is defined. So in this case, it's attempting to look for the file in the project hosting the Compliance Yaml instead of the project running the pipeline.

This MR now solves this problem by allowing the user to specify exactly the context to search in. And because it supports expanding variables in the subkeys, we can now do this in the compliance Yaml:

include:
  - project: '$CI_PROJECT_PATH'
    file: '$CI_CONFIG_PATH'
    ref: '$CI_COMMIT_SHA'
    rules:
      - exists:
          paths:
            - '$CI_CONFIG_PATH'
          project: '$CI_PROJECT_PATH'
          ref: '$CI_COMMIT_SHA'

Testing steps:

1. Ensure the FF is ON.
2. Create a new blank project named My Project. In this project:

• Go to Settings > CI/CD > Auto DevOps and ensure "Default to Auto DevOps pipeline" is NOT enabled.

3. Create a new blank project named Compliance Project. In this project:

• Create the following files:

compliance.yml:

# Old configuration that breaks the pipeline on projects without a configuration file
include:
  - project: '$CI_PROJECT_PATH'
    file: '$CI_CONFIG_PATH'
    ref: '$CI_COMMIT_SHA'
    rules:
      - exists: # Checks in the current file's context: compliance-project on `main`. It sees that `.gitlab-ci.yml` exists and attempts to include the project's config file, but it's not there. So it breaks the pipeline.
          - '$CI_CONFIG_PATH'

compliance-job:
  script: echo

.gitlab-ci.yml:

# We just need this file to exist in the Compliance Project

4. In the group in which you created the earlier projects:

• Go to Settings > General.
• Expand the Compliance frameworks section.
• In Compliance pipeline configuration (optional), add a new framework with name My Compliance with the path to the compliance Yaml: e.g. compliance.yml@group-e/compliance-project

5. In My Project, now apply the compliance framework label.

• Go to Settings > Compliance framework. Select and apply My Compliance.

6. In My Project, go to Pipelines and run a new pipeline. Observe that you get the following error:

7. Now update compliance.yml with the following:

# New configuration that does not break the pipeline
include:
  - project: '$CI_PROJECT_PATH'
    file: '$CI_CONFIG_PATH'
    ref: '$CI_COMMIT_SHA'
    rules:
      - exists: #  Checks in the pipeline project's context: my-project on `main`. It sees that it does not have a config file, so it does not attempt to include it.
          paths:
            - '$CI_CONFIG_PATH'
          project: '$CI_PROJECT_PATH'
          ref: '$CI_COMMIT_SHA'

compliance-job:
  script: echo

8. In My Project, go to Pipelines and run a new pipeline. Observe that now the pipeline runs successfully, with only the compliance-job in the pipeline.

Screenshot

Related to #386040 (closed)