Multiple pipelines in multiple files MVC
Note: This is not a priority, just capturing thoughts from several different issues into a single proposal.
I read somewhere that all CI/CD systems are either: individual jobs, pipelines, or graphs. GitLab CI is clearly "pipelines" and trying to do graphs at all might just be wrong. But having said that:
- People want to be able to have multiple pipelines that are completely independent, but described in separate files.
Docs ⟼⬤⟶⬤ Android ⟼⬤⟶⬤⟶⬤ iOS ⟼⬤⟶⬤⟶⬤
master ⟼⬤⟶⬤⟶⬤ feature branches ⟼⬤⟶⬤
- People want to be able to have a CI pipeline that must pass before a separate CD pipeline can run, and want to describe them in separate files.
CI ⟼⬤⟶⬤ ➘ CD ⬤⟶⬤⟶⬤
- People want to be able to do complex CI/CD configurations that include trees, or parallel branches that should run independently, branched from different points.
⟼⬤⟶⬤⟶⬤ | ↳⬤⟶⬤ ↳⬤⟶⬤⟶⬤
Perhaps we can stop short of taking a generalized "graph" approach, and think instead of multiple "pipelines", with some dependencies between them (perhaps leveraging dependencies we need to support cross-project pipelines). By either forward or backward reference to one pipeline triggering another, and supporting multiple pipelines running concurrently, we effectively have created a backdoor for enabling this kind of complex behavior, while still keeping our "pipeline" core.
In terms of implementation, it's possible to define these pipelines in a single
.gitlab-ci.yml file, but as stated above, people want separate files. A high-leverage way to handle that would be to
include separate pipeline definitions into the master
.gitlab-ci.yml (https://gitlab.com/gitlab-org/gitlab-ce/issues/20868). But perhaps there's some justification for supporting them in completely separate files. One such reason is to have different
.gitlab-ci.yml in sub-directories, which only apply to files changed in those sub-directories. Intuitively, I'd want logically-separate pipelines declared in separate files.
Monorepo use case
For monorepos, users would want to check in a yaml for each component in the monorepo. The expected behavior would be to traverse the tree up from a check-in until we find a yaml, and use it for execution.
- Allow declarations of multiple (named?) pipelines (optionally in separate files, via includes or pointers or ???)
- Allow declaring regular conditions for triggering pipelines such as
- Allow declaring file-based conditions for triggering pipeline such as file or directory regex (pushes may trigger multiple pipelines with matching regexes)
- Allow declaring pipeline-based dependencies to trigger pipeline (e.g. run pipeline X iff pipeline Y succeeds)
- Allow declaring job-based dependencies to trigger pipeline (e.g. run pipeline X iff job A in pipeline Y succeeds)
Note: There may be a natural parallel between pipeline/job dependencies and artifact dependencies, but since sometimes artifacts are managed outside of our artifact system (such as docker images, environment state, or runner state), these dependencies need to work with or without artifacts.
Links / references
- Make it possible to define multiple pipelines: #22972 (moved)
- Multi-project pipelines: https://gitlab.com/gitlab-org/gitlab-ee/issues/933
- Import/include common CI/CD content from one project into another's
.gitlab-ci.yml: #20868 (closed)
- Parallel pipelines for different environments of a project: #19471 (closed)
- Some additional context: https://gitlab.com/gitlab-org/gitlab-ce/issues/25457#note_23649819
- Run jobs only/except when there are modifications on a given path: https://gitlab.com/gitlab-org/gitlab-ce/issues/19232
- Enable directed acyclic graphs using
pipelineto include w/ local variables: https://gitlab.com/gitlab-org/gitlab-ce/issues/56214