Consider fetching the gitlab assets from security instead of dev
Context
Generating a coordinated pipeline also generates a tag on Omnibus and CNG:
These tags are created on our dev instance (dev.gitlab.org) because we normally use dev to build everything. The Omnibus pipeline includes the fetch_assets
job, that searches for the assets generated on the GitLab dev project. For example, in the above case, the fetch_assets
job depends on this gitlab-ee pipeline https://dev.gitlab.org/gitlab/gitlab-ee/-/pipelines/233151, particularly on the build-assets-image
job. Summarized the dependency goes like this:
- Coordinated pipeline on Ops depends on
- Omnibus pipeline on dev that depends on
- GitLab pipeline on dev
- Omnibus pipeline on dev that depends on
The problem
Pipelines on the GitLab dev project are not thoroughly observed, unless someone is keeping an eye on the #master_broken_mirrors
channel it usually takes some time to notice pipelines on dev are failing. In the most recent example, we had an MR that changed the CI configuration making it incompatible with the dev instance gitlab-org/gitlab!84378 (comment 903120062). This wasn't noted at the time of merging because the changes were compatible with the gitlab.com
instance. This blocked the coordinated pipelines for some hours until that MR was reverted.
Proposal
Let's fetch the assets from Security instead of dev. For reference the build-assets-image
job is also executed on the GitLab security project (https://gitlab.com/gitlab-org/security/gitlab/-/pipelines/512398535 / https://gitlab.com/gitlab-org/security/gitlab/-/jobs/2310826290). Some advantages:
- GitLab security pipelines are usually faster than the ones on dev
- When something fails on GitLab security, it's noted more promptly because that prevents auto-deploy packages from being tagged.
- We no longer rely on the CI configuration of the gitlab-ee dev project.
Another idea could be to also make sure changes like gitlab-org/gitlab!84378 (merged) are tested against dev instances but this might belong to Engineering productivity.