Dynamic child pipeline creation via artifact includes
Problem to solve
include keyword (https://docs.gitlab.com/ee/ci/yaml/#include) currently supports 4 keywords:
We don't support including a file from an artifact, though, which would be useful for child/parent pipelines (#16094 (closed)) in order to generate CI YAML in one job and dynamically start a pipeline using that YAML in a subsequent one.
Feature will be rolled out behind
ci_dynamic_child_pipeline feature flag.
Let's suppose we have a job that generates pipeline configurations and we want child pipelines to be created using the dynamically generated configs:
stage: - build - test generate-config: stage: build script: - generate_yaml --service=A > config-a.yml - generate_yaml --service=B > config-b.yml artifacts: paths: - config-a.yml - config-b.yml service-a: stage: test trigger: include: - artifact: config-a.yml job: generate-config service-b: stage: test trigger: include: - artifact: config-b.yml job: generate-config
trigger:include:artifactconfig entry and validations
- enable server-side artifact download when
- allow creation of child pipeline to evaluate
- One possibility is that if the
jobkeyword is not provided, it will search through all artifacts available to the job and use the first matching file.
- Use a
bootstrapjob that where all trigger jobs can download artifacts from
bootstrap: script: - generate_yaml >my_1.yml - generate_yaml >my_2.yml artifacts: paths: - my_1.yml - my_2.yml use_yaml_trigger_1: trigger: include: - artifact: my_1.yml # defaults to taking artifact from bootstrap job, as bootstrap is specified implicitly as `job: bootstrap` use_yaml_trigger_2: trigger: include: - artifact: my_2.yml job: bootstrap # optional since the generating job is called bootstrap
We default to a job called
bootstrap if no
job: value is provided. This would make things easier in simple use cases.
Constraints and limitations
artifact keyword would not be usable on the top-level as follows, since that would result in undefined behavior (no job has run at the point the top-level
include keyword is parsed):
include: - artifact: my.yml job: generate_yaml
Permissions and Security
As trigger jobs are executed server-side the
include:artifact causes a download of the artifact on the server in order to be evaluated. We need to treat this carefully from security and performance point of view.
An important example in the documentation to provide for this feature should be setting up matrix builds using child pipelines.