### Problem to solve <!-- What problem do we solve? --> As pipelines grow more complex, a few related problems start to emerge: - The staged structure where all steps in a stage must be completed before the first job in next stage begins causes arbitrary waits, slowing things down - Configuration for the single global pipeline becomes very long and convoluted, making it hard to understand - Imports exacerbate the above item, and create the potential for namespace collisions where jobs are unintentionally duplicated - Pipeline UX can become unwieldy with so many jobs and stages to work with Need statement: **The user** needs a way to **let jobs not unnecessarily depend on other jobs within the same pipeline context** so that they can **let parallel job trails continue without being blocked by each other**. ### Intended users <!-- Who will use this feature? If known, include any of the following: types of users (e.g. Developer), personas, or specific company roles (e.g. Release Manager). It's okay to write "Unknown" and fill this field in later. Personas can be found at https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/ --> [Devon (DevOps Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#devon-devops-engineer) ### Further details <!-- Include use cases, benefits, and/or goals (contributes to our vision?) --> ### Proposal <!-- How are we going to solve the problem? Try to include the user journey! https://about.gitlab.com/handbook/journeys/#user-journey --> If a parent (originating) pipeline was able to trigger a set of concurrently running child pipelines, you could solve each of these problems: - Child pipelines would execute each of their jobs still according to a stage sequence, but would be free to continue forward through their stages without waiting for unrelated jobs to finish. - Configuration would be distributed out into each of the child pipeline configurations, reducing cognitive load to understand everything. - Imports would be done at the child pipeline level, reducing the likelihood of collisions - Each pipeline would have only the steps relevant, making it easier to understand what's going on. You also get some nice benefits for doing things this way: - By using existing triggering functionality, you can take advantage of `only: changes` type keywords to trigger pipelines only when certain files change (this is valuable for monorepos, for example). - By keeping the base (parent) `.gitlab-ci.yml` as a normal pipeline, it can have its own behaviors and sequencing in relation to triggers. - Also by keeping it a normal pipeline, if someone doesn't use this feature, it just works exactly as you'd expect. No special configuration. - By taking advantage of status attribution (https://gitlab.com/gitlab-org/gitlab-ee/issues/11238), the pipeline can wait for success of the child without any other special code/configuration required, wait for it to complete but not care about result, or can just trigger it and not follow it at all. All of this will work with includes, so you can retain composability within the configuration. This first issue will allow for one level of child pipelines (i.e., one parent, n pipelines) and child pipelines will not be able to trigger further downstream pipelines. We will address this in a fast-follow issue (https://gitlab.com/gitlab-org/gitlab-ce/issues/63566) where we will allow for some additional levels, with som limit to prevent infinite recursion. #### Implementation Introduce a new syntax for triggering a child pipeline by pointing to a configuration yml within the repository: ```yaml microservice_a: trigger: local: microservice_a/config.yml strategy: depend only: changes: - microservices_a/**/* ``` For this example, a triggering job called `microservice_a` in the parent pipeline would be triggered that would use the yaml in `microservice_a/config.yml` as its configuration. It would only run if there are changes in the `microservice_a` folder, and [treat it as a dependency](https://gitlab.com/gitlab-org/gitlab-ee/issues/11238) (i.e., fail if the child pipeline fails.. alternatively, the `wait` strategy would wait for it to finish, but not care if it passes or fails.) A pipeline triggered in this way would start running independently but would have status attribution to a parent pipeline (via https://gitlab.com/gitlab-org/gitlab-ee/issues/11238). If you look at the parent pipeline's pipeline page, you should see that the pipeline is running or completed, and if you click on it, it should take you to the child pipeline's pipeline page. By implementing things in this way, we do need to radically change the user experience within GitLab and it works largely in the way you'd expect. There is always a parent pipeline since child pipelines can only be triggered from a parent. - `Latest` tag will not show on child pipelines - Child pipelines will receive a dedicated tag indicating that they are child pipelines #### Real-life Example GitLab QA and gitlab-org/gitlab-qa#6, where we would like to trigger pipeline for GitLab CE or GitLab EE checks depending on project that triggering MR belongs to. This may be also interesting feature for GitLab Omnibus, to build EE / CE images easier. #### Considerations 1. Child pipelines should not contribute to `Commit Status`, 1. Child pipelines should not be taken for Merge Request status (again, only the parent pipeline matters but since it has status attribution this should be fine), 1. Likely we should show child pipelines as part of `Commit#Pipelines` and `Merge Request#Pipelines`, 1. Likely we should consider a design that shows parent pipeline, and allows you to "expand" parent to see also child pipelines, 1. `variables:` of trigger job should be passed to the child pipeline allowing to fine-tune it, 1. Child pipeline should inherit all settings of parent pipeline when running `trigger: local:`? ### Future Enhancement - For the MVC, a parent can trigger child pipelines, but child pipelines cannot trigger further child pipelines. This could be opened up in the future potentially. - Add support for multiple pipelines in different files (https://gitlab.com/gitlab-org/gitlab-ce/issues/28592) - or do we get this for "free" with include/extends given the above implementation? - Filtering of child pipelines (able to hide them or hide them by default) will not be supported with the mvc ### Permissions and Security <!-- What permissions are required to perform the described actions? Are they consistent with the existing permissions as documented for users, groups, and projects as appropriate? Is the proposed behavior consistent between the UI, API, and other access methods (e.g. email replies)? --> ### Documentation <!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html Add all known Documentation Requirements here, per https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements --> ### Testing <!-- What risks does this change pose? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? See the test engineering process for further guidelines: https://about.gitlab.com/handbook/engineering/quality/guidelines/test-engineering/ --> ### What does success look like, and how can we measure that? <!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. --> ### Links / references - [Powerful, integrated primitives vision](https://about.gitlab.com/direction/cicd/#theme-powerful-integrated-primitives) * `pipeline` to include w/ local variables: https://gitlab.com/gitlab-org/gitlab-ce/issues/56214 * Slack Channel (Internal Link only) [#f_child_par_pipelines](https://gitlab.slack.com/app_redirect?channel=f_child_par_pipelines) has been defined.