### Problem to solve The `include` keyword (https://docs.gitlab.com/ee/ci/yaml/#include) currently supports 4 keywords: - `local` - `file` - `template` - `remote` We don't support including a file from an artifact, though, which would be useful for child/parent pipelines (https://gitlab.com/gitlab-org/gitlab/issues/16094) in order to generate CI YAML in one job and dynamically start a pipeline using that YAML in a subsequent one. ### Intended users Pipeline authors ### Further details Feature will be rolled out behind `ci_dynamic_child_pipeline` feature flag. ### Proposal Let's suppose we have a job that generates pipeline configurations and we want child pipelines to be created using the dynamically generated configs: ```yaml stage: - build - test generate-config: stage: build script: - generate_yaml --service=A > config-a.yml - generate_yaml --service=B > config-b.yml artifacts: paths: - config-a.yml - config-b.yml service-a: stage: test trigger: include: - artifact: config-a.yml job: generate-config service-b: stage: test trigger: include: - artifact: config-b.yml job: generate-config ``` 1. define `trigger:include:artifact` config entry and validations 1. enable server-side artifact download when `trigger:include:artifact` is used 1. allow creation of child pipeline to evaluate `trigger:include:artifact` by consequence #### Future improvements - One possibility is that if the `job` keyword is not provided, it will search through all artifacts available to the job and use the first matching file. - Use a `bootstrap` job that where all trigger jobs can download artifacts from ```yaml bootstrap: script: - generate_yaml >my_1.yml - generate_yaml >my_2.yml artifacts: paths: - my_1.yml - my_2.yml use_yaml_trigger_1: trigger: include: - artifact: my_1.yml # defaults to taking artifact from bootstrap job, as bootstrap is specified implicitly as `job: bootstrap` use_yaml_trigger_2: trigger: include: - artifact: my_2.yml job: bootstrap # optional since the generating job is called bootstrap ``` We default to a job called `bootstrap` if no `job:` value is provided. This would make things easier in simple use cases. ### Constraints and limitations The `artifact` keyword would not be usable on the top-level as follows, since that would result in undefined behavior (no job has run at the point the top-level `include` keyword is parsed): ```yaml include: - artifact: my.yml job: generate_yaml ``` ### Permissions and Security As trigger jobs are executed server-side the `include:artifact` causes a download of the artifact on the server in order to be evaluated. We need to treat this carefully from security and performance point of view. ### Documentation <!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html Add all known Documentation Requirements here, per https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements If this feature requires changing permissions, this document https://docs.gitlab.com/ee/user/permissions.html must be updated accordingly. --> An important example in the documentation to provide for this feature should be setting up matrix builds using child pipelines. ### Testing <!-- What risks does this change pose? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? See the test engineering process for further help: https://about.gitlab.com/handbook/engineering/quality/test-engineering/ --> ### What does success look like, and how can we measure that? <!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. --> ### What is the type of buyer? <!-- Which leads to: in which enterprise tier should this feature go? See https://about.gitlab.com/handbook/product/pricing/#four-tiers --> ### Links / references