Dynamic child pipeline creation via artifact includes
Problem to solve
The include
keyword (https://docs.gitlab.com/ee/ci/yaml/#include) currently supports 4 keywords:
local
file
template
remote
We don't support including a file from an artifact, though, which would be useful for child/parent pipelines (#16094 (closed)) in order to generate CI YAML in one job and dynamically start a pipeline using that YAML in a subsequent one.
Intended users
Pipeline authors
Further details
Feature will be rolled out behind ci_dynamic_child_pipeline
feature flag.
Proposal
Let's suppose we have a job that generates pipeline configurations and we want child pipelines to be created using the dynamically generated configs:
stage:
- build
- test
generate-config:
stage: build
script:
- generate_yaml --service=A > config-a.yml
- generate_yaml --service=B > config-b.yml
artifacts:
paths:
- config-a.yml
- config-b.yml
service-a:
stage: test
trigger:
include:
- artifact: config-a.yml
job: generate-config
service-b:
stage: test
trigger:
include:
- artifact: config-b.yml
job: generate-config
- define
trigger:include:artifact
config entry and validations - enable server-side artifact download when
trigger:include:artifact
is used - allow creation of child pipeline to evaluate
trigger:include:artifact
by consequence
Future improvements
- One possibility is that if the
job
keyword is not provided, it will search through all artifacts available to the job and use the first matching file. - Use a
bootstrap
job that where all trigger jobs can download artifacts from
bootstrap:
script:
- generate_yaml >my_1.yml
- generate_yaml >my_2.yml
artifacts:
paths:
- my_1.yml
- my_2.yml
use_yaml_trigger_1:
trigger:
include:
- artifact: my_1.yml # defaults to taking artifact from bootstrap job, as bootstrap is specified implicitly as `job: bootstrap`
use_yaml_trigger_2:
trigger:
include:
- artifact: my_2.yml
job: bootstrap # optional since the generating job is called bootstrap
We default to a job called bootstrap
if no job:
value is provided. This would make things easier in simple use cases.
Constraints and limitations
The artifact
keyword would not be usable on the top-level as follows, since that would result in undefined behavior (no job has run at the point the top-level include
keyword is parsed):
include:
- artifact: my.yml
job: generate_yaml
Permissions and Security
As trigger jobs are executed server-side the include:artifact
causes a download of the artifact on the server in order to be evaluated. We need to treat this carefully from security and performance point of view.
Documentation
An important example in the documentation to provide for this feature should be setting up matrix builds using child pipelines.