Dynamic child pipeline creation via artifact includes

Problem to solve

The include keyword (https://docs.gitlab.com/ee/ci/yaml/#include) currently supports 4 keywords:

• local
• file
• template
• remote

We don't support including a file from an artifact, though, which would be useful for child/parent pipelines (#16094 (closed)) in order to generate CI YAML in one job and dynamically start a pipeline using that YAML in a subsequent one.

Intended users

Pipeline authors

Further details

Feature will be rolled out behind ci_dynamic_child_pipeline feature flag.

Proposal

Let's suppose we have a job that generates pipeline configurations and we want child pipelines to be created using the dynamically generated configs:

stage:
  - build
  - test
generate-config:
  stage: build
  script:
    - generate_yaml --service=A > config-a.yml
    - generate_yaml --service=B > config-b.yml
  artifacts:
    paths:
      - config-a.yml
      - config-b.yml

service-a:
  stage: test
  trigger:
    include:
      - artifact: config-a.yml
        job: generate-config

service-b:
  stage: test
  trigger:
    include:
      - artifact: config-b.yml
        job: generate-config

1. define trigger:include:artifact config entry and validations
2. enable server-side artifact download when trigger:include:artifact is used
3. allow creation of child pipeline to evaluate trigger:include:artifact by consequence

Future improvements

• One possibility is that if the job keyword is not provided, it will search through all artifacts available to the job and use the first matching file.
• Use a bootstrap job that where all trigger jobs can download artifacts from

bootstrap:
  script:
    - generate_yaml >my_1.yml
    - generate_yaml >my_2.yml
  artifacts:
    paths:
      - my_1.yml
      - my_2.yml

use_yaml_trigger_1:
  trigger:
    include:
      - artifact: my_1.yml # defaults to taking artifact from bootstrap job, as bootstrap is specified implicitly as `job: bootstrap`

use_yaml_trigger_2:
  trigger:
    include:
      - artifact: my_2.yml
        job: bootstrap # optional since the generating job is called bootstrap

We default to a job called bootstrap if no job: value is provided. This would make things easier in simple use cases.

Constraints and limitations

The artifact keyword would not be usable on the top-level as follows, since that would result in undefined behavior (no job has run at the point the top-level include keyword is parsed):

include:
  - artifact: my.yml
    job: generate_yaml

Permissions and Security

As trigger jobs are executed server-side the include:artifact causes a download of the artifact on the server in order to be evaluated. We need to treat this carefully from security and performance point of view.

Documentation

An important example in the documentation to provide for this feature should be setting up matrix builds using child pipelines.

Testing

What does success look like, and how can we measure that?

What is the type of buyer?

Links / references