Enforce required stages in pipeline execution policies

What does this MR do and why?

Add support for enforcement of required stages in Pipeline execution policies (Epic: #475152 (closed)). Placement of required stages can be defined in the policy YAML and the stages become available in the main pipeline. Example policy CI:

name: Inject project CI
description: ''
enabled: true
pipeline_config_strategy: inject_ci
required_stages:
- name: ".pipeline-policy-test"
  after: ['test', 'build', '.pipeline-policy-pre']
- name: ".pipeline-policy-build"
  after: ['build']
- name: ".pipeline-policy-deploy"
  after: ['deploy', '.pipeline-policy-test']
content:
  include:
  - project: gitlab-org/pipeline-execution-policies/compliance-project
    file: override.yml
    ref: main

Background

• We're seeing the need from the customers to use custom stages. For example:
  • #494431 (comment 2142853678)
  • #475152 (comment 2098708853)
  • #475152 (comment 2127499404)
• The need for custom stages is amplified by the fact that with override_project_ci, no custom stages can be used. Any custom stages that could be used with inject_ci where they can be defined in project CI are ignored in override_project_ci. We ignore the project CI to prevent any invalid or missing configuration from running the pipelines. By ignoring the project CI, we only have option to utilize default and reserved stages for override_project_ci strategy.
• We explored some stage injection in the scope of the PoC in !150156 (comment 1885060158) and !151061 (closed), but decided against it and only went forward with the reserved stages because it was unclear how to inject and merge stages from multiple pipelines.

Considerations

• I propose to define required_stages in the policy YAML instead of trying to use each pipeline's stages to merge them together. The concept of merging stages from multiple pipelines comes from the design of Pipeline execution policies and is not tied to the CI YAML.
• Using pipeline's stages doesn't provide enough information on how / where to inject a stage. There could be collisions between multiple policy pipelines, and the project pipeline.
• Stage placement is defined using after keyword, which defines preference for the stage placement. If a policy is enforced on a group level and there are many projects with different configurations, this gives policy makers some flexibility to say (as an example): "I want to add this stage after test, but if test doesn't exist, use build".
• In the Pipeline execution design / PoC, we were considering .pipeline-policy-test stage which would have implicit fallback [test, build, .pre]. With after keyword, users can be explicit about the stage placement.
• after keyword is required to avoid any "magic" behavior
• If policy makers define after conditions in such a way that is incompatible, the pipeline fails:
  • after: [custom] but custom stage is not present in the project pipeline
  • there are cyclic dependencies within the required stages: [{ name: 'deploy', after: ['build', 'test'] }, { name: 'build', after: ['deploy'] }]
• If required_stages define a stage in a different place than .gitlab-ci.yml, the stage is rearranged according to the conditions in required_stages. For example: required_stages: [{ name: 'deploy', after: ['build', 'test'] } and stages: [build, deploy, test], the result is stages: [build, test, deploy].
• If multiple policies define the same custom stage, the most top-level group's placement configuration has priority over project-level policies.
• If a top-level group specifies a policy stage, it becomes available for all sub-group and project policies, they don't have to respecify it. It only needs to be specified it in the policy CI configuration in stages (see the nuances below)

Caveats

• This design may not provide full stage order enforcement for inject_ci strategy. Theoretically, the development team can decide to define a custom project stage where they deploy the application before compliance jobs run, for example with stages: [build, custom-deploy, test, deploy]. However:
  • Crucial checks could still be enforced in .pipeline-policy-pre to avoid unexpected surprises
  • If customers need this level of control, they could use override_project_ci.

Nuances

• required_stages definition in the policy YAML only allows the stage to be used in the policy CI configuration YAML. The policy CI configuration still needs to define any custom stage in stages. This duplication could lead to some confusion. We could solve it by implicitly allowing any policy_stage in the same way as we allow the reserved stages, but this is not implemented yet in this PoC.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Group policy config:

name: Group policy
description: ''
enabled: true
pipeline_config_strategy: inject_ci
required_stages:
- name: ".pipeline-policy-test"
content:
  include:
  - project: gitlab-org/pipeline-execution-policies/compliance-project
    file: group.yml
    ref: main

Project policy config:

name: Project policy
description: ''
enabled: true
pipeline_config_strategy: inject_ci
required_stages:
- name: ".pipeline-policy-build"
  after:
  - build
- name: ".pipeline-policy-deploy"
  after:
  - deploy
  - ".pipeline-policy-test"
content:
  include:
  - project: gitlab-org/pipeline-execution-policies/compliance-project
    file: override.yml
    ref: main

Before (custom stages ignored)	After	With invalid after conditions

How to set up and validate locally

1. Enable the feature flag

   Feature.enable(:custom_pipeline_execution_policy_stages)

2. Create a project
3. Create a policy using the examples above, specifying a non-default stage in required_stages
4. Create a pipeline and verify that stage is injected in the correct place

Related to #475152 (closed)