As you may know, Debian is using gitlab, which means we are improving the distribution because of it. But we would like to keep pushing forward. Therefore, on behalf of the Debian project, we would like to request release this feature as part of the Community Edition to keep improving the Debian distribution quality.
Currently, we have a pipeline definition that allows us to execute the same tests that are executed after a package is uploaded to Debian but on every commit. Please check the README.md for more info.
Use case
In Debian, we have thousands of projects where the output are Debian packages. A very common case (not to say almost always), a Debian package has other packages that depend on it. So, the idea is testing before doing an upload to the Debian archive if the next release of the package doesn't break the packages that depend on it.
Build dependencies
PackageA build depends on PackageB. A new release of PackageB can break PackageA on building time (in Debian a.k.a. Fails To Build From Source - FTBFS)
Runtime dependencies
PackageA depends on PackageB for running. A new release of PackageB can break in runtime PackageA e.g. a library release can break a binary application, a new python module can break an application that depends on it.
Per our buyer-based model, triggering pipelines via automation is something that an IC does rather than somebody's manager or director. This can also already be worked around by using API calls, so we aren't really gating anything except adding a manual step.
We don't have a plan at the moment to make this feature free. Your workaround should work, though - I believe we've had other users write a small script or program to handle the triggering and waiting.
Hi @jlenny, we are here at debconf19@curitiba and this topic came up a lot of times so far. We would like to have a better answer from gitlab because we are trying to get as much adoption as we can (package maintainers using CI) and this feature is becoming in something really important for the Debian project.
There were a lot of workarounds proposed but none of them is similar to the feature that gitlab already has :(.
@sytses pinging you, just in case you want to take a look at it and give some feedback.
Hey everyone, I agree that this feature can move to core, based on our buyer-based pricing model (triggering a pipeline from one project to another is a common task done by an individual contributor.) @sfwgitlab please let me know if you disagree with this pricing decision or feel like we should escalate further.
+1 from me because the lack of multi project pipelines encourages bundling of disjunct code units in single projects which is an anti pattern and harmful to good coding practices in general. Bringing this to Core would help improve software quality in a lot of projects, ultimately benefitting everyone.
I'd also like to know about the reasons behind this decision. We've working to prepare our workflow at Debian for this feature to be enabled and this is a big drawback.
@petermarko I'd like to clarify, the blog was not an announcement. It was us sharing our planned roadmap. Other companies tend to keep 2 roadmaps, an internal one, and a more conservative, customer-facing roadmap. At GitLab we believe in transparency and share our roadmap not just with customers, but with everyone publicly. We work from the same roadmap that we share. This means that our roadmap can and does change more frequently than other companies because we are showing you the same thing we work from in the interest of transparency.
Given the confusion, I think in this case it sounds like we weren't clear about our level of certainty and or descriptive of how we operate in the open. I'm sorry it didn't have more clear language and, as a reviewer for our blog, I'll be sure to watch for this type of potential confusion in the future so we can be transparent, but also clear.
It feels a little bit unfair coming back here after you made the decisions and not explaining at all what happened in the middle. But I prefer to think you made a mistake closing this issue.
@jlenny maybe my explanation in the issue wasn't perfect but I already mentioned that functionality and it is not equivalent for Debian at all. That was the reason why I decided opening this issue.
Not sure what kind of value adds to your premium customers, but for Debian it is something really necessary. We can't afford having one job per pipeline triggered waiting for the result.
@agustinhenze which feature are you referring to specifically? In the review process, we discussed the proposal section of this document which was to move the trigger keyword from paid to free. A docs pointer or .gitlab-ci.yml example might help. Is it possible that https://gitlab.com/gitlab-org/gitlab-ee/issues/11238, which will come out on the 22nd, was the feature you wanted moved to core?
@jlenny As I understand from here, we need to use the variable CI_JOB_TOKEN for the visualization, which is not available on community edition according to here.
Mhm, guess @jlenny might want to reconsider the decision :D FWIW I am somewhat fine with not having "trigger:" as YAML construct in .gitlab-ci.yml, but without at least the UI support to display the whole pipeline this is messy as hell.
@apollo13 I just tried your .gitlab-ci.yml and it showed the downstream pipeline for me. Perhaps you've run into a bug.
Another challenge here is that we seem to be all talking about a few different features. This issue description (and decision) was very explicitly about the trigger keyword, but the consensus now seems to be evolving that the trigger keyword was actually not that critical, and there are 2-3 other features that different folks are looking to move to free (or, perhaps, are already also free).
@jlenny Are you testing against a free or premium system ;) Also could you share the setup setups (create trigger tokens or not) and your gitlab-ci file?
@jlenny Is this a contradiction to your statement above about availability in all tiers?
@apollo13 do you have a silver or gold account on gitlab.com, how does it work there?
Anyways, at least for the core edition, I see no workaround for multi-project pipelines. In my tests I faced a 401 error with CI_JOB_TOKEN, which confirms the statement by @petermarko above.
In my tests I faced a 401 error with CI_JOB_TOKEN, which confirms the statement by @petermarko above.
Interestingly enough I got a 404 with CI_JOB_TOKEN. If I override it in CI variables to be an actual Trigger value it works. Wonder where the discrepancy between 404 and 401 comes from.
We'll go back to the drawing board here and validate which of these things work or not. The intention was that the trigger keyword was a paid feature, hence the overall feature being listed there, and curling was a workaround that was functionally equivalent but not as syntactically nice. It seems that there are some cases where different related features are working on different tiers (or not.) Please let me know if I've missed any part of the feature set to validate:
trigger keyword: Should be paid (the workaround is to use curl on the API instead)
Visualizing downstream pipelines: Should be free/core, but is gated by requiring you to curl or use trigger to interact with the feature
Hasn't been confirmed yet by anyone here, but depend/wait strategies is a related feature that should also work, whichever way you trigger the pipeline. https://gitlab.com/gitlab-org/gitlab-ce/issues/66990 for follow up.
@matteeyah can you please confirm in the code that the above statements are true?
Triggering jobs with CI_JOB_TOKEN should be free/core
Yes please, because it does not seem to be.
Visualizing downstream pipelines: Should be free/core, but is gated by requiring you to curl or use trigger to interact with the feature
How does this "gating" look for curl. Ie could I have a helper docker image which provides a few bash functions that you can do "trigger my_group/my_project" which then in turn uses curl to fetch the project id and do the triggering? Or do you really mean to require the use of curl literally in a script line of the ci file.
@apollo13 yes, literally just that it requires you to either have curled in your script or used the trigger syntax. That's the intended current state. This issue would have also made the trigger keyword free, as per the proposal section of the description.
It seems the gating for visualization occurs via the CI_JOB_TOKEN. curl with a personal API token would also trigger the pipeline, but cannot visualize it.Otherwise the description looks great. Thanks @jlenny!
@apollo13 I've tested again in a private project on .com, which is free tier and your .gitlab-ci.yml behaved as expected. It would seem to be a bug where the feature is available in core but not free.
EDIT: Yeah, I think you found the underlying difference @fennibay
CI_JOB_TOKEN is 100% available in core - see Ci::Build#L481.
It looks like triggering pipelines with a trigger is available in core, but not triggering with a CI_JOB_TOKEN - according to Ci::PipelineTriggerService. It was originally added to core with https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2017, but it somehow ended up in the paid version.
Hasn't been confirmed yet by anyone here, but depend/wait strategies is a related feature that should also work, whichever way you trigger the pipeline. https://gitlab.com/gitlab-org/gitlab-ce/issues/66990 for follow up.
Given that depend/wait only make sense when triggering a pipeline via a multi-project setup, is it worth to maintain the two codepaths (trigger in gitlab-ci.yml + support for curl) instead of pushing trigger to core? I am asking purely form a code maintenance standpoint. Also I'll wait curiously on how wait is going to work with curl and intermedient network hickups :D
Hey everyone (incl. @agustinhenze), wanted to follow up on this request. Since we last checked in, the bug #31573 (closed) (parity between trigger/curl) has been resolved, which if I recall correctly was the main blocker for open source workflows. Want to confirm is that the case, and if so, can this issue be closed?
trigger is more than syntactical sugar and there is no way to implement the "wait" functionality without it:
it is bad when the syntax of a computer language differs between paid and unpaid variants; scripts will suddenly fail with YAML parsing errors when you use them on a system with a smaller license.
waiting within a job is no option because this blocks the builder (e.g. what happens when there is only one builder or all other builders are waiting too?). Starting of downstream builds must be done by gitlab without occupying a builder. This is not possible without special syntax (the trigger: keyword).
Everyone, given the feedback above I'm going to go ahead and close this issue. That's not a permanent action, though, so if you come across this issue later and have some objection please ping me as I'd love to hear your feedback. Thanks in the meantime to all those who provided feedback here and helped us resolve the bug that was impacting most of you.