Merge `stages:` across includes

Problem to solve

When including a template with stages, the behavior is not as one would expect: the stages key from the included template overwrites rather than merges with the including .gitlab-ci.yml. So, if you have a .gitlab-ci.yml which defines the following stages.

stages:
  - build
  - deploy
  - test

build_job:
  stage: build
  ...

And then include a file that defines a new compliance stage:

stages:
  - compliance

compliance_job:
  stage: compliance

The pipeline will fail because the resulting pipeline only has a compliance stage, rather than a build, deploy, test, and compliance stage. Specifically, these (very common) scenarios are problematic in the current implementation:

1. If the include defines a stages key, existing pipelines will break, as YAML validation will fail if pipelines define jobs that use stage names not defined in the include.
2. If the include does not define a stages key (or does not define stage key for its jobs) pipelines will fail YAML validation if the developer pipelines define a stages key.

Intended users

The intended users for this iteration are GitLab system administrators and users responsible for implementing enforcement of governance policies for CI pipelines within GitLab.

Proposal

Allow the inclusion of stages from enforced templates in an additive manner, so that stages in an enforced template are combined with stages defined in other pipelines.

Referring to the example above, the resulting file would end up with build, deploy, test, and compliance stages rather than only compliance. This behaves much more as you'd expect if you include a template that introduces a new compliance stage.

Permissions and Security

This should follow existing permissions for includes.

Documentation

Testing

We should run the manual CI job package-and-qa on any MR for this feature as this end-to-end test should validate if Auto DevOps is working properly.

There are edge cases in how stages can be defined that should be carefully validated. Specifically, those edge cases are

1. If no stages are defined in .gitlab-ci.yml, then the build, test and deploy are allowed to be used as job’s stage by default.
2. If a job doesn’t specify a stage, the job is assigned the test stage.

The implementation of this proposal should consider that existing pipelines that fall into the above two edge cases may be broken by the inclusion of a required stages key.

What does success look like, and how can we measure that?

The implementation of this proposal is successful if a user is able to add additional stages via an include and apply it to existing pipelines with varied existing configurations without requiring any changes to existing .gitlab-ci.yml configurations in individual developer repositories.

What is the type of buyer?

Links / references

Relevant conversation thread from gitlab-ee#8429

GitLab CI yaml documentation

MRs that originally implements the feature: gitlab-ee!14011 and gitlab-ee!13923 (FE)

Group-level required pipelines: gitlab-ee#11343

&762

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.