Making use of multi-project pipelines for triggers
Ever since multi-project pipelines were introduced in GitLab, we wanted to use it for our triggered pipelines for our triggered QA runs (package-and-qa
in GitLab projects). However, ability to run a pipeline against master (protected branch) is controlled by ability to merge/push to master. This meant, for developers to trigger a pipeline in omnibus-gitlab, they needed to be able to run a pipeline against master in omnibus-gitlab. Due to security reasons and implications, we can not open up our master branch to all the developers.
This was raised in gitlab-qa#63 (closed), and the workaround of "Open up master to developers but enfore MR approvals to ensure nothing gets merged without approvals" isn't acceptable to us, essentially because it is a sub-optimal workaround both from an engineering perspective and from security perspective (< Insert door being locked by cheetos meme >). Hence we opened gitlab#24585 to split this access, but from the looks of it, that issue is not being scheduled anytime soon.
Due to these reasons, we decided not to (or was unable to) use multi-project pipelines. We added scripts to trigger pipelines as a bot user and hacked it to pass necessary information between pipelines, do polling-based-waiting mechanism to get the status and workarounds like that.
I think we should dogfood multi-project pipelines, and this is a proposal to do that. Or at least do a bit of research around that.
Proposal
- Mirror
omnibus-gitlab
toomnibus-gitlab-mirror
(like we did with CNG and CNG-mirror). - Open up
omnibus-gitlab-mirror
's master branch to developers to merge. - Mirror
gitlab-qa
togitlab-qa-mirror
. - Open up
gitlab-qa-mirror
's master branch to developers to merge. - Alter CI configuration of
omnibus-gitlab
so the triggered pipeline runs in mirror while the regular test pipeline runs in original. - Alter CI configuration of
gitlab
to drop the use of custom scripts, but use multi-project pipeline to trigger pipelines againstomnibus-gitlab-mirror
. - Alter CI configuration of
omnibus-gitlab
to drop the use of custom scripts, and trigger pipelines inomnibus-gitlab-mirror
andgitlab-qa-mirror
using multi-project pipelines.
Pros
- Dogfooding
- Proper attribution of pipelines to the people who started it, instead of everything being owned by a bot user.
- Easy access to the downstream pipelines from upstream MR widgets, thanks to multi-project pipelines.
- De-clutter the pipeline page of
omnibus-gitlab
. - Triggered pipeline in
omnibus-gitlab
fromomnibus-gitlab
MR no longer hiding the spec pipelines (This is because the MR widget by default shows the latest pipeline against the commit, and this will be the triggered pipeline)
Cons
- Duplication of projects to its mirrors.
@gitlab-org/distribution @gitlab-org/quality Interested to know your opinions here.