MR pipeline support in Sec templates
Release notes
Problem to solve
MR pipelines are a feature that a number of customers are interested in, however, delivering that feature across all of the Sec templates has proven difficult. As it is a significant change, attempts have been made, and failed during a few different milestones. This issue should be used to coordinate how we might deliver this, including alternate approaches.
Some context for the most recent discussions that led to not delivering this in %16.0 can be found here and here.
Reminder: CI/CD Templates are deprecated and that will need to be taken into consideration when delivering this work.
Additional details
MR pipeline support has been added to the .latest
templates and the remaining blocker to transition the stable template to them is the AutoDevOps support. Indeed, some security features (ex: Container Scanning, DAST, and Fuzzing jobs) have dependencies on the Build
job, which doesn't run in MR pipelines.
Good summary from Lucas: #410880 (comment 2152506266)
Some part of the overall problem to solve can be addressed with other approaches (like allowing Scan Execution Policies to use the .latest
template) but eventually some features really require to support MR pipelines to provide a relevant experience. That's the case for Secret Detection for instance, which uses MR pipelines enablement as a trigger to scan all commits in the MR diff rather than only scanning the head of the branch.
Proposal
See original proposals
1. Use CI/CD components
Handle MR pipelines support in CI/CD components only. Do not touch existing CI/CD templates.
- Pros:
- No risk of impacting existing users who don't want to use MR pipelines
- Incentivize to migrate over CI/CD components to benefit from newer features
- No breaking change. CI templates are not changed and existing CI/CD components that require an update can release a new Major version without impacting users of current version.
- Cons:
- Customers have to migrate from CI templates to CI/CD components to benefit from MR pipelines support.
- CI/CD components are not mature enough for a viable UX in self managed environments. &15224 (comment 2283373852).
- Scan Execution Policies do not work with CI/CD components. This requires work not currently defined and planned. #489904 (comment 2172295365)
- All security features must provide CI/CD components. #410880 (comment 2287739349)
- All security features might need to align how their CI/CD components are implemented vs rules and MR pipelines.
- Maintains discrepancy between the "latest" and "stable" CI templates.
branch
template for ADO
2. Use CI templates, and add dedicated Update the rules
in the existing "stable" CI templates (similar to what we already have in latest
templates) to provide the MR pipeline support. Additionally, create new branch
CI templates to mimic old behavior and use these in ADO to maintain existing support for branch only usage.
- Pros:
- All existing customers of Sec feature will benefit from the change without any manual action.
- Provide MR pipeline support with Scan Execution Policies
- Keeps ADO support as it is (do not break features depending on
build
job).
- Cons:
- Risk of impacting all existing customer, including these who don't use MR pipelines.
- Long term maintenance cost for additional "branch only" template
- No MR pipeline support in ADO (consistent accross all features)
TODO: Would this be considered a breaking change?
=> Yes. Some customers might have done some changes in their CI config after enabling MR pipelines to ensure some jobs still run in branch pipeline too, to fulfill the needs for some of our jobs (e.g keeping the build job for CS to work in branch pipeline). If we flip the CS job to support running in MR pipelines, then their custom setup might break.
3. Use CI templates, and remove ADO support when necessary
Same as proposal 2, but remove ADO support for features no longer compatible with MR pipeline in this context rather than adding dedicated branch
templates. Compatible features are kept in ADO and provide MR pipeline support.
- Pros:
- All existing customers of Sec feature will benefit from the change without any manual action
- Provides MR pipeline support with Scan Execution Policies
- MR pipeline support in ADO for compatible features
- Cons:
- Risk of impacting all existing customers, including these who don't use MR pipelines.
- BREAKING CHANGE: Removes ADO support for some features (introduce discrepancy).
4. other alternatives to review:
- Adjust MR widget, MR comparison logic and policies to consider branch pipelines too as source of scan results? Let's imagine we don't run Security scans at all in CI jobs, but in the backend or an external service once a branch or an MR is created. Their execution would not be tied to the configuration of MR pipeline or branch pieline. The creation of a pipeline or an MR would trigger a scan execution based on the content of that branch. So we might actually don't really care in which type of pipeline the security scan ran? As far as a scan has run for a given ref, should it be relevant to be considered?
- While this could alleviate some of the problems, this still won't be enough to cover some usages. For instance, Secret Detection must run in an MR pipeline setup to scan all commits in MRs, rather than just the head of the branch.
- Make Scan Execution Policies able to work with
.latest
templates. Technically doable, but could tightly couple unrelated concepts and increase the risk of breaking SEPs which are usually leveraged by large enterprise customers who expect reliability. -> this has actually already been implemented with Expand Scan Execution Policies to run on MR pip... (#415427 - closed)
From the original propoals above (expand to see them), and after a synchronous discussion (2025-01-13) between DRIs (@connorgilbert @mbenayoun @gonzoyumo) we've decided the following:
- Proposal 1, using CI/CD components is rejected. This would not offer the expected customer results, requires too much work, and represents too much risk due to its current maturity level and it's current usage.
- Proposal 2 and 3, using CI templates and either adding
branch
templates and/or (partially) dropping ADO support offers the best path forward. Deciding the details will depend on further investigation and discussion with the impacted groups. - Suggested alternatives are not sufficient to provide the expected customer results.
UPDATE 2025-01-29
Further testing the proposal, we've realized that the current behavior of the latest
template is actually enforcing AST CI/CD jobs to run in an MR pipelines. This means users who have their jobs running in a regular pipeline when an MR is created will face the duplicate pipeline issue. See #410880 (comment 2320969911).
As a result the proposal is updated to consider an opt-in approach instead of the default enablement we have with the latest
template. By checking an additional environment variable before enabling MR pipeline support in our CI/CD jobs, the stable
templates would look like this:
rules:
# Add the job to merge request pipelines if MR Pipelines for AST are enabled and there's an open merge request.
- if: $AST_ENABLE_MR_PIPELINES == "true" && $CI_PIPELINE_SOURCE == "merge_request_event"
# Don't add it to a *branch* pipeline if MR Pipelines for AST are enabled and there's an open merge request.
- if: $AST_ENABLE_MR_PIPELINES == "true" && $CI_OPEN_MERGE_REQUESTS
when: never
# Add the job to branch pipelines.
- if: $CI_COMMIT_BRANCH
This approach allows us to modify the stable
CI/CD templates with:
- no impact for existing users of the stable templates, particularly if they don't want to migrate to MR pipelines
- no impact for ADO support, as the default behavior is still to run in a branch pipeline
- no impact on SEP. If users don't want MR pipelines, they just leave the stable template without any change
This means that there is no deprecation announcement to be made in 17.9 as there will be no breaking changes in 18.0.
If users of Scan Execution Policies wants to enable MR pipelines, they will be able to use the stable
template if they configure the additional environment variable or they can continue to use the latest
templates.
Here is an example MR simulating how a user would enable MR pipelines for AST features: gitlab-org/secure/tests/olivier/mr-pipelines-testing/existing-with-mr-pipelines!3 (diffs)
TODO: validate the possibility to implement this proposal before 18.0. If there is no breaking change, nothing prevents us from shipping the change today.
TODO: consider if/when to align latest
templates with stable
templates
TODO: we may decide to take the inverse approach for Secret Detection, and have this particular feature enabling the MR pipeline by default. NB: if we want to do this, we can't simply set the shared variable to true at the global level in the SD template, otherwise it will enable it for all other included security features. It has been decided that SD will follow the same opt-in approach.
TODO: validate Merge Results pipelines too. See #410880 (comment 2321629558)
Next steps & timeline
- By 2025-01-17: Verify using ADO with MR pipelines is actually a thing that makes sense. E.g. if the build job does not support MR pipelines, is ADO at all working then? Is it worth trying to get our features working in ADO with MR pipelines enabled?
- By 2025-01-17: Devise an implementation plan and planning breakdown.
- By 2025-01-24: Gather feedback and get approval on the implementation plan from impacted group's leader/DRI and Sec leadership?
-
2025-01-29: Feedback from testing lead to an updated proposal - By 2025-31: Test and validate the updated proposal. Get approval from group's leader.
- By 2025-02-07 Implement the necessary changes that must ship in 17.9 (announcement, doc update, etc.)
Implementation plan
See former proposal implementation plan
-
Each group validate the proposal for its feature(s). Please review MR pipelines in AutoDevOps and Behavior of each AST feature in the context of AutoDevOps to the necessary context
- Code Quality (@connorgilbert): make no changes because CodeClimate-based scanning is scheduled for removal in %18.0 (#471677).
- SAST (@connorgilbert): add MR pipeline support to the "stable" CI template in 18.0 and keep it included in AutoDevOps.
- IaC (@connorgilbert): add MR pipeline support to the "stable" SAST-IaC CI template in 18.0 and keep it included in AutoDevOps (if it is there in the first place).
- Secret Detection (@abellucci): add MR pipeline support to the "stable" CI template in 18.0 and keep it included in AutoDevOps.
- Dependency Scanning (@johncrowley): add MR pipelines support to the "stable" CI template in 18.0 and keep it included in AutoDevOps.
-
Container Scanning (@johncrowley): Pick one option of these options
-
add a specific
Container-Scanning.branch.gitlab-ci.yml
template with no support for MR pipelines and use it in ADO. Then add MR pipelines support to the stable CI template. - Remove Auto Container Scanning feature from AutoDevOps (remove the inclusion of the CS template), and add MR pipelines support to the stable CI template.
-
Do the necessary work to add MR pipelines support to
Auto Build
, then add MR pipelines support to the stable CI template. - Other suggestion?
-
add a specific
-
DAST (@johncrowley): Pick one option of these options
-
add a specific
DAST.branch.gitlab-ci.yml
template with no support for MR pipelines and use it in ADO. Then add MR pipelines support to the stable CI template. - Remove Auto DAST feature from AutoDevOps (remove the inclusion of the DAST template), and add MR pipelines support to the stable CI template.
-
Do the necessary work to add MR pipelines support to
Auto Build
andAuto Deploy
, then add MR pipelines support to the stable CI template. - Other suggestion?
-
add a specific
-
Update all stable templates to add the option to enable MR pipeline
- Pending MR: !179652
- Update the documentation
-
TBC: update
latest
template to align with the opt-in approach?