Backend: Running a pipeline in a project with an external config file and Compliance Framework config

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

This issue derives from the error described in Pipeline fails when project includes another pr... (#393596 - closed). The error appears to be the result of an unaccounted use case related to the cross-functionality of the Compliance Framework and the CI/CD configuration file. The purpose of this issue is to bring this use case to light and discuss whether a feature change is necessary. --> We have determined that a solution to support this use case is necessary.

The Use Case

In a project that has a custom (external) CI/CD configuration file and Compliance Framework label applied, the customer would like the pipeline to run both the Compliance YAML as well as the external config YAML.

Why is it Needed?

A customer noted that they use an external CI/CD configuration file to prevent users from editing the config YAML and thereby potentially exposing the values of the project's variables. Using protected branches/variables is not a viable option as they would like the variables to be available for pipelines on non-protected branches. Given this set up, they would like the Compliance Framework to accommodate running an external config file.

Background

Compliance Framework

Per documentation:

You can create a compliance framework that is a label to identify that your project has certain compliance requirements or needs additional oversight. The label can optionally enforce compliance pipeline configuration to the projects on which it is applied.

When you choose to enforce the compliance config, the project that you apply a Compliance Framework label to runs the Compliance YAML instead of the project's own CI/CD config file. However, you can run both if you include the project's config file in the Compliance YAML.

Custom CI/CD Configuration File

A project defaults to using the root's .gitlab-ci.yml for the pipeline config. However you can change this setting and even point it to an external config file as explained in Specify a custom CI/CD configuration file.

Example

  • Project 1 contains the Compliance Framework XYZ YAML: .base-compliance.yml. This YAML is configured to include the target project's CI/CD configuration file as such:
include:
  - project: '$CI_PROJECT_PATH'
    file: '$CI_CONFIG_PATH'
    ref: '$CI_COMMIT_SHA'
  • Compliance Framework XYZ label is applied to Project 3 which has an external CI/CD configuration file located in Project 2. Project 3's config_path is thus set to: .gitlab-ci.yml@group-1/project-2
  • The user now runs the pipeline in Project 3 which is forced to run the Compliance Framework YAML. The following error occurs:
Included file `.gitlab-ci.yml@group-1/project-2` does not have YAML extension!`

You can follow the steps here to reproduce the error.

The Problem

The .base-compliance.yml's include configuration is not designed to handle a situation where the target project's CI/CD configuration file is external.

From a code perspective, the error is caused by:

  • The values of $CI_PROJECT_PATH and $CI_COMMIT_SHA are of the target project (Project 3) instead of the project that contains the CI/CD configuration file (Project 2)
  • $CI_CONFIG_PATH is populated with the project's full config path which is .gitlab-ci.yml@group-1/project-2, and this syntax is not supported by include. This results in the aforementioned pipeline error.

Other Workarounds Considered

  1. Use trigger instead of include in the Compliance YAML.

--> Why it doesn't work:

  • It has similar issues to include and could cause a circular pipeline situation.
  1. Update Project 3's internal config file to have the following, and host the variables on Project 2.
include:
  - project: 'group-1/project-2'
    file: '.gitlab-ci.yml'

--> Why it doesn't work:

  • If the variables are hosted on Project 3, then anyone could edit the config file with echo and expose the variable values.
  • If the variables are hosted on Project 2, the variables are no longer available to Project 3's pipeline.
  1. Trigger a multi-project pipeline (to Project 2) in Project 3.

--> Why it doesn't work:

  • If the variables are hosted on Project 3, then anyone could edit the config file with echo and expose the variable values.
  • If the variables are hosted on Project 2, the variables are available to the pipeline but then it runs within the context of Project 2's repo.
  1. Implement a sort of pseudo-compliance framework where you update the external config file in Project 2 to be something like:
include: .base-compliance.yml
...
(Rest of config)

And then restrict the access to Project 2.

--> Why it doesn't work:

  • It defeats the point of having the Compliance Framework label feature.
  1. Use protected branches and protected variables as noted here.

--> Why it doesn't work:

  • The project's variables must be available to pipelines on non-protected branches as well. See further details here.

Current Workaround

Update the Compliance YAML's include statement with the following. (The PROTECTED_* variables can be renamed as necessary.)

include:
  # If the custom path variables are defined, include the project's external config file 
  - project: '$PROTECTED_PIPELINE_CI_PROJECT_PATH'
    file: '$PROTECTED_PIPELINE_CI_CONFIG_PATH'
    ref: '$PROTECTED_PIPELINE_CI_REF'
    rules:
      - if: $PROTECTED_PIPELINE_CI_PROJECT_PATH && $PROTECTED_PIPELINE_CI_CONFIG_PATH && $PROTECTED_PIPELINE_CI_REF
  # If any custom path variable is not defined, include the project's internal config file as normal
  - project: '$CI_PROJECT_PATH'
    file: '$CI_CONFIG_PATH'
    ref: '$CI_COMMIT_SHA'
    rules:
      - if: $PROTECTED_PIPELINE_CI_PROJECT_PATH == null || $PROTECTED_PIPELINE_CI_CONFIG_PATH == null || $PROTECTED_PIPELINE_CI_REF == null

In each project that requires the Compliance Framework and has an external CI/CD configuration file, specify the values for all three PROTECTED_* variables.

This workaround has been added to the docs here.

Potential Solutions / Ideas

Idea 1: Update include:file to accept new syntax and override project and ref values

Adapt the current recommended configuration:

include:
  - project: '$CI_PROJECT_PATH'
    file: '$CI_CONFIG_PATH'
    ref: '$CI_COMMIT_SHA'

such that include:file supports the @ syntax. The project path and reference value that it extracts from $CI_CONFIG_PATH (the file value) would override the values of include:project and include:ref.

The idea here is to do a quick fix on the part that's throwing the error.

--> Concerns:

  • This is a "hack"-ish fix and would not align with current product/UX expectations.

Idea 2: Feature to enable running both the Compliance YAML and project's config file

Add the ability to configure the project to run both the Compliance YAML and the project's config YAML. The idea here is that we would combine the YAMLs internally instead of requiring the customer to use the include config. Or we could run them in separate pipelines (triggered internally).

--> Concerns:

  • Requires a new feature implementation that may need cross-domain coordination.

Idea 3: Update the include keyword to support a full path string

Update the include keyword to support a full path string that would be internally parsed and identified as one of: local, project, or remote path. Ref: https://docs.gitlab.com/ee/ci/yaml/#include

The formats supported would be the same ones supported when specifying a Custom CI/CD configuration file, which are:

  • Local:
    • .my-custom-file.yml
    • my/path/.my-custom-file.yml
  • Project:
    • .gitlab-ci.yml@namespace/another-project
    • my/path/.my-custom-file.yml@namespace/sub-group/another-project
    • my/path/.my-custom-file.yml@namespace/sub-group1/sub-group2/another-project:refname
  • Remote:
    • https://gitlab.com/group/project/remote.gitlab-ci.yml

Shorthand usage:

include: my/path/.my-custom-file.yml@namespace/sub-group1/sub-group2/another-project:refname
  • NOTE: The above format is currently an alternative syntax for local, but we want to update it to support the new full path value which can be either local, project, or remote.

Normalized usage:

include:
  - full: my/path/.my-custom-file.yml@namespace/sub-group1/sub-group2/another-project:refname

Further notes:

  • The template path will not be included in this support and will have to remain explicitly specified with the template subkey per docs.
  • We will need to support parsing the @project/path and :refname values from a Project full path.
  • We will need to update the include documentation with the new full path support.
  • Ensure the entire Example configuration section (including subsections) of the Compliance Framework documentation is updated. The new recommended way to include the project's configuration within the Compliance Framework YAML should be:
include:  # Execute individual project's configuration (supports an internal or external config file)
  - full: '$CI_CONFIG_PATH'

--> Concerns:

  • We would be introducing a new path syntax that is inconsistent with the FDQN syntax for include:component.

Idea 4: Implement FQDN syntax include:fqdn

Enable FQDN syntax for the path and internally parse it to determine the file type.

--> Concerns:

  • Complicated implementation.

Idea 5: Define new path variables

We can define variables as described in #393960 (comment 1391750093):

  • CI_CONFIG_PATH_FILE
  • CI_CONFIG_PATH_PROJECT
  • CI_CONFIG_PATH_REF

These will be defined unless the external configuration is a remote URL. And the user's config in the description can be updated to be:

include:
  - project: '$CI_CONFIG_PATH_PROJECT'
    file: '$CI_CONFIG_PATH_FILE'
    ref: '$CI_CONFIG_PATH_REF'

--> Concerns:

Proposal

[2023-06-22] We want to investigate a more holistic reimplementation of compliance framework pipelines that could solve all of the challenges together.

A new issue has been created that consolidates the list of challenges we are aware of so far: #416104 (closed)

[2023-06-20] We are currently investigating Idea 2: Feature to enable running both the Compliance YAML and project's config file. The main idea is to trigger both the Compliance and Project config files as separate child pipelines. This would be an opt-in functionality.

Goals:

  • The Project pipeline should run regardless if the config file is external or remote.
  • The Project pipeline should run without triggering a circular pipeline scenario in which the Compliance pipeline is triggered again.
  • The Compliance pipeline should always run first and pass before the Project pipeline is run.
  • The Compliance pipeline should run even if the Project does not have a CI config file configured.
  • The user should be able to define top-level global keywords (such as stages:) in both the Compliance and Project config files.

POC: Draft: POC for triggering Compliance and Projec... (!124231 - closed)

Edited by 🤖 GitLab Bot 🤖