Backend: Running a pipeline in a project with an external config file and Compliance Framework config
Summary
This issue derives from the error described in Pipeline fails when project includes another pr... (#393596 - closed). The error appears to be the result of an unaccounted use case related to the cross-functionality of the Compliance Framework and the CI/CD configuration file. The purpose of this issue is to bring this use case to light and discuss whether a feature change is necessary. --> We have determined that a solution to support this use case is necessary.
The Use Case
In a project that has a custom (external) CI/CD configuration file and Compliance Framework label applied, the customer would like the pipeline to run both the Compliance YAML as well as the external config YAML.
Why is it Needed?
A customer noted that they use an external CI/CD configuration file to prevent users from editing the config YAML and thereby potentially exposing the values of the project's variables. Using protected branches/variables is not a viable option as they would like the variables to be available for pipelines on non-protected branches. Given this set up, they would like the Compliance Framework to accommodate running an external config file.
Background
Compliance Framework
Per documentation:
You can create a compliance framework that is a label to identify that your project has certain compliance requirements or needs additional oversight. The label can optionally enforce compliance pipeline configuration to the projects on which it is applied.
When you choose to enforce the compliance config, the project that you apply a Compliance Framework label to runs the Compliance YAML instead of the project's own CI/CD config file. However, you can run both if you include
the project's config file in the Compliance YAML.
Custom CI/CD Configuration File
A project defaults to using the root's .gitlab-ci.yml
for the pipeline config. However you can change this setting and even point it to an external config file as explained in Specify a custom CI/CD configuration file.
Example
-
Project 1 contains the Compliance Framework
XYZ
YAML:.base-compliance.yml
. This YAML is configured to include the target project's CI/CD configuration file as such:
include:
- project: '$CI_PROJECT_PATH'
file: '$CI_CONFIG_PATH'
ref: '$CI_COMMIT_SHA'
- Compliance Framework
XYZ
label is applied to Project 3 which has an external CI/CD configuration file located in Project 2. Project 3'sconfig_path
is thus set to:.gitlab-ci.yml@group-1/project-2
- The user now runs the pipeline in Project 3 which is forced to run the Compliance Framework YAML. The following error occurs:
Included file `.gitlab-ci.yml@group-1/project-2` does not have YAML extension!`
You can follow the steps here to reproduce the error.
The Problem
The .base-compliance.yml
's include
configuration is not designed to handle a situation where the target project's CI/CD configuration file is external.
From a code perspective, the error is caused by:
- The values of
$CI_PROJECT_PATH
and$CI_COMMIT_SHA
are of the target project (Project 3) instead of the project that contains the CI/CD configuration file (Project 2) -
$CI_CONFIG_PATH
is populated with the project's full config path which is.gitlab-ci.yml@group-1/project-2
, and this syntax is not supported by include. This results in the aforementioned pipeline error.
Other Workarounds Considered
- Use
trigger
instead ofinclude
in the Compliance YAML.
--> Why it doesn't work:
- It has similar issues to
include
and could cause a circular pipeline situation.
- Update Project 3's internal config file to have the following, and host the variables on Project 2.
include:
- project: 'group-1/project-2'
file: '.gitlab-ci.yml'
--> Why it doesn't work:
- If the variables are hosted on Project 3, then anyone could edit the config file with
echo
and expose the variable values. - If the variables are hosted on Project 2, the variables are no longer available to Project 3's pipeline.
- Trigger a multi-project pipeline (to Project 2) in Project 3.
--> Why it doesn't work:
- If the variables are hosted on Project 3, then anyone could edit the config file with
echo
and expose the variable values. - If the variables are hosted on Project 2, the variables are available to the pipeline but then it runs within the context of Project 2's repo.
- Implement a sort of pseudo-compliance framework where you update the external config file in Project 2 to be something like:
include: .base-compliance.yml
...
(Rest of config)
And then restrict the access to Project 2.
--> Why it doesn't work:
- It defeats the point of having the Compliance Framework label feature.
- Use protected branches and protected variables as noted here.
--> Why it doesn't work:
- The project's variables must be available to pipelines on non-protected branches as well. See further details here.
Current Workaround
Update the Compliance YAML's include statement with the following. (The PROTECTED_*
variables can be renamed as necessary.)
include:
# If the custom path variables are defined, include the project's external config file
- project: '$PROTECTED_PIPELINE_CI_PROJECT_PATH'
file: '$PROTECTED_PIPELINE_CI_CONFIG_PATH'
ref: '$PROTECTED_PIPELINE_CI_REF'
rules:
- if: $PROTECTED_PIPELINE_CI_PROJECT_PATH && $PROTECTED_PIPELINE_CI_CONFIG_PATH && $PROTECTED_PIPELINE_CI_REF
# If any custom path variable is not defined, include the project's internal config file as normal
- project: '$CI_PROJECT_PATH'
file: '$CI_CONFIG_PATH'
ref: '$CI_COMMIT_SHA'
rules:
- if: $PROTECTED_PIPELINE_CI_PROJECT_PATH == null || $PROTECTED_PIPELINE_CI_CONFIG_PATH == null || $PROTECTED_PIPELINE_CI_REF == null
In each project that requires the Compliance Framework and has an external CI/CD configuration file, specify the values for all three PROTECTED_*
variables.
This workaround has been added to the docs here.
Potential Solutions / Ideas
include:file
to accept new syntax and override project
and ref
values
Idea 1: Update Adapt the current recommended configuration:
include:
- project: '$CI_PROJECT_PATH'
file: '$CI_CONFIG_PATH'
ref: '$CI_COMMIT_SHA'
such that include:file
supports the @
syntax. The project path and reference value that it extracts from $CI_CONFIG_PATH
(the file
value) would override the values of include:project
and include:ref
.
The idea here is to do a quick fix on the part that's throwing the error.
--> Concerns:
- This is a "hack"-ish fix and would not align with current product/UX expectations.
Idea 2: Feature to enable running both the Compliance YAML and project's config file
Add the ability to configure the project to run both the Compliance YAML and the project's config YAML.
The idea here is that we would combine the YAMLs internally instead of requiring the customer to use the include
config. Or we could run them in separate pipelines (triggered internally).
--> Concerns:
- Requires a new feature implementation that may need cross-domain coordination.
include
keyword to support a full path string
Idea 3: Update the Update the include
keyword to support a full path string that would be internally parsed and identified as one of: local
, project
, or remote
path. Ref: https://docs.gitlab.com/ee/ci/yaml/#include
The formats supported would be the same ones supported when specifying a Custom CI/CD configuration file, which are:
- Local:
.my-custom-file.yml
my/path/.my-custom-file.yml
- Project:
.gitlab-ci.yml@namespace/another-project
my/path/.my-custom-file.yml@namespace/sub-group/another-project
my/path/.my-custom-file.yml@namespace/sub-group1/sub-group2/another-project:refname
- Remote:
https://gitlab.com/group/project/remote.gitlab-ci.yml
Shorthand usage:
include: my/path/.my-custom-file.yml@namespace/sub-group1/sub-group2/another-project:refname
-
NOTE: The above format is currently an alternative syntax for
local
, but we want to update it to support the new full path value which can be eitherlocal
,project
, orremote
.
Normalized usage:
include:
- full: my/path/.my-custom-file.yml@namespace/sub-group1/sub-group2/another-project:refname
Further notes:
- The
template
path will not be included in this support and will have to remain explicitly specified with thetemplate
subkey per docs. - We will need to support parsing the
@project/path
and:refname
values from a Project full path. - We will need to update the
include
documentation with the new full path support. - Ensure the entire Example configuration section (including subsections) of the Compliance Framework documentation is updated. The new recommended way to include the project's configuration within the Compliance Framework YAML should be:
include: # Execute individual project's configuration (supports an internal or external config file)
- full: '$CI_CONFIG_PATH'
--> Concerns:
- We would be introducing a new path syntax that is inconsistent with the FDQN syntax for
include:component
.
include:fqdn
Idea 4: Implement FQDN syntax Enable FQDN syntax for the path and internally parse it to determine the file type.
--> Concerns:
- Complicated implementation.
Idea 5: Define new path variables
We can define variables as described in #393960 (comment 1391750093):
- CI_CONFIG_PATH_FILE
- CI_CONFIG_PATH_PROJECT
- CI_CONFIG_PATH_REF
These will be defined unless the external configuration is a remote URL. And the user's config in the description can be updated to be:
include:
- project: '$CI_CONFIG_PATH_PROJECT'
file: '$CI_CONFIG_PATH_FILE'
ref: '$CI_CONFIG_PATH_REF'
--> Concerns:
- Remote path is not supported.
- Incompatibility between projects that use an external config file versus ones that use an internal config file. It may require additional
rules:if
conditions to accommodate. Details: #393960 (comment 1411809453) #393960 (comment 1412988899)
Proposal
[2023-06-22] We want to investigate a more holistic reimplementation of compliance framework pipelines that could solve all of the challenges together.
A new issue has been created that consolidates the list of challenges we are aware of so far: #416104
[2023-06-20] We are currently investigating Idea 2: Feature to enable running both the Compliance YAML and project's config file. The main idea is to trigger both the Compliance and Project config files as separate child pipelines. This would be an opt-in functionality.
Goals:
- The Project pipeline should run regardless if the config file is external or remote.
- The Project pipeline should run without triggering a circular pipeline scenario in which the Compliance pipeline is triggered again.
- The Compliance pipeline should always run first and pass before the Project pipeline is run.
- The Compliance pipeline should run even if the Project does not have a CI config file configured.
- The user should be able to define top-level global keywords (such as
stages:
) in both the Compliance and Project config files.
POC: Draft: POC for triggering Compliance and Projec... (!124231 - closed)