Nested pipelines (nested gitlab-ci.yml) to support monorepos
Problem to solve
Integrated/single-view CI for independent pipelines in a monorepo.
GitLab promotes and delivers significant value to its users with a single application based on convention over configuration. (handbook/strategy/goals)
A lot of the advantages of a single application directly or indirectly relate to the organization of the codebase.
At GitLab we have shown in recent times that building the single application in a single repository has some problems. This has lead to parts of that repository forking off into smaller repositories that are packaged separately. I think we have the ability to ease some of these problems for ourselves and others, and return to some of the advantages of the single application (ehem, repository, I don't mean to conflate the two so boldly but I believe there is a strong relationship)
One of these problems is the complexity and size of CI. This was one of the presented reasons for a new repo for gitlab-ui. Moving our widely shared UI components into a separate repo allowed for a clean sheet for CI/CD. Multiple versions could be tested and released independently. The total pipeline is much quicker and cheaper. This has given the maintainers space to add more tooling that harnesses CI to provide a better developer experience.
Intended users
- Delaney (Development Team Lead)
- Sasha (Software Developer)
- Devon (DevOps Engineer)
Particularly useful for teams working on monorepos.
Potentially useful for GitLab.
Proposal
It would be great if we could use monorepo tools like lerna and have independent CI pipelines for each package. Without any configuration beyond providing gitlab-ci.yml files to define your pipelines.
- The nested pipelines would only be executed if there are changes in its correlating package.
- The root pipeline would not be executed if there are only changes in the individual packages
- The root pipeline is executed if any of the root source is changed, which would include any version bumps of the contained packages.
I recognize this could be possible with a single gitlab-ci.yml, but I think that moves most of the problem to the yaml file, which isn't ideal. We can provide a better experience and a clearer definition of these individual pipelines.
What does success look like, and how can we measure that?
Independent CI pipeline definitions in a single repository, a single view for these pipelines for MRs, commits, so on...
Links / references
https://about.gitlab.com/handbook/product/single-application/
https://about.gitlab.com/handbook/product/#convention-over-configuration
https://about.gitlab.com/company/strategy/#goals
https://github.com/lerna/lerna
Example monorepo packages
https://github.com/lerna/lerna/tree/master/commands
https://github.com/facebook/jest/tree/master/packages
(more at https://lernajs.io/#users)