Compliance Pipelines to Pipeline Execution Policy Variable Migration issue
Summary
As part of migrating from Compliance Pipelines to Pipeline Execution Policy, Customers have reported a bug / blocker for migration.
-
When using Compliance Pipeline we are able to define / enforce both Compliance & Project Variables from CI file.
(in UI before triggering pipeline for a branch)
-
But are unable to do so when using Pipeline Execution Policy (Post migration from existing Compliance pipeline CI file)
Steps to reproduce
- Create a Compliance Pipeline with Variables in Compliance Pipeline and Project Pipeline using Compliance Framework
PROJECT_VAR:
value: "a"
options:
- "a"
- "b"
- "c"
- Apply Compliance Framework over a Project (Where Compliance Pipeline exist)
- Go To Project > Pipelines > New Pipeline in Project
- Notice the CI Variables getting auto-populated in New Pipeline UI
- Migrate the Compliance Pipeline to Pipeline Execution Policy
- Go To Project > Pipelines > New Pipeline in Project where PEP was implemented with same logic
- Trigger the Pipeline and notice missing CI Variables as shown in screenshots above
Example Project
- Project with Compliance Pipeline: https://gitlab.com/rmsrohan-ultimate/test-bank/sample-project-pipeline-variables/-/pipelines/new
- Notice the CI Variables getting Auto-Populated.
- Project with Pipeline Execution Policy: https://gitlab.com/rmsrohan-ultimate/test-bank/sample-project-pep-pipeline-variables/-/pipelines/new
- Notice the CI Variables that should Auto-Populated missing.
What is the current bug behavior?
Missing CI Variables when migrating from Compliance Pipelines to Pipeline Execution Policy
What is the expected correct behavior?
When migrating from Compliance Pipeline to Pipeline Execution Policy the CI Variables should load and the behaviour should remain the same.
Relevant logs and/or screenshots
Added in above Summary
Output of checks
Possible fixes
We would need to implement support for prefilled variables for PEP.
As far as I understand, the prefilled variables are fetched like this:
- By invoking Ci::ListConfigVariablesService which uses
YamlProcessor
to parse the CI config - It has no context of pipeline execution policies and only parses the project config
- As this service is invoked via GQL from the PipelineVariablesForm, it wouldn't be performant enough to parse all PEP configs in addition to the project config
I think we could make it work by extending our "PEP config analyzer" which is currently parsing the enforced scanners from PEP configs. We have a BBM planned for %18.0 to backfill data. We can:
- extend the service now to parse out the prefilled variables and store them in the DB
- the BBM will take care of backfilling this data for all existing PEPs
- extend Ci::ListConfigVariablesService to merge PEP prefilled variables from the DB for all PEPs applicable to the project.