Multi project pipeline
Description
We have three types of pipelines:
- CI pipeline: Multiple stages, one project, similar to naming in Jenkins, planning to build 8.8
- Deploy pipeline: The flow of deploying code to servers through various stages
- Project pipeline: Cross-project dependencies, particularly for micro-services, but also for complicated build dependencies: e.g. api -> front-end, ce/ee -> omnibus. Like https://ci.concourse.ci/
The first two types have received a lot of work lately; this issue covers the third. We call this the Project Pipeline in our vision doc.
We have basic triggers, but need to improve them, including adding first-class support for them in .gitlab-ci.yml
. We also need dependencies that you define in .gitlab-ci.yml per project. All-multi project visualizations can be based on this; you need read access to the other projects. You don't have one uber pipeline defined somewhere; projects depend on each other to define the pipeline. We can show the project pipeline even in the project view without needing an entity to represent the group of pipelines.
Proposal
-
First-class triggers -
Cross-project dependencies -
Cross-project artifacts -
Link between project pipeline views -
Consolidated view of entire pipeline across projects -
Use Docker image registry and Docker Compose to run cross-project integration tests within single project's pipeline
MVP
Just do API to make things possible, add visualization.
-
Trigger API using job token (https://gitlab.com/gitlab-org/gitlab-ee/issues/2611) -
Artifact API using job token (https://gitlab.com/gitlab-org/gitlab-ee/issues/2770) Stretch -
Link visualization, not consolidated one (https://gitlab.com/gitlab-org/gitlab-ee/issues/2121) -
Wrapper for curl call? - .gitlab-ci.yml template example in blog post Stretch -
Treat pipelines with different inputs (e.g. variables) as distinct? (https://gitlab.com/gitlab-org/gitlab-ce/issues/25734)
Using CI_JOB_TOKEN
with existing APIs makes it easier to trigger other pipelines and download artifacts from other pipelines, but not exactly "first class". But it also allows us to recognize the source of the job token, and thus internally tie these pipelines together, so we can start visualizing the relationships (gitlab-ce#22550). This let's us move quickly to make things "possible" today, and go back and make them "easy" later (https://gitlab.com/gitlab-org/gitlab-ce/issues/16556 and/or #1681 (moved)). An even simpler "easier" implementation is to provide some kind of wrapper for the curl
calls.
This also lets us experiment with proper failure attribution and blocking. e.g. if project X triggers project Y, and Y's CI fails, attribute that error all the way back to project X's pipeline, and block related merge requests from being merged.
Use Cases
- Push change to API topic branch, trigger tests in front-end to still pass, failures attributed to API.
API ⟼⬤⟶⬤
➘
Front-end ⬤⟶⬤
- Push change to API master, trigger tests in front-end to still pass, bundle both into package, release. Failures attributed to API. (Although there's an argument for attributing failures to packager, if package needs to be updated with approved API changes.)
API ⟼⬤⟶⬤
➘
Front-end ⬤⟶⬤⟶⬤
➘
Package ⬤⟶⬤
-
Push change to Front-end topic branch, pull artifacts from API master, failures attributed to Front-end.
-
Push change to Front-end topic branch which depends on API topic branch, pull artifacts from API topic branch, failures attributed to Front-end.