Add status checking behaviors to pipeline triggers
Problem to Solve
Currently our implementation of multi-project pipelines is quite limited. One of the most important things that are missing is a feedback mechanism. We can trigger an external pipeline using a multi-project pipelines feature, but the problem is that a triggering pipeline does not wait for the external pipeline status, thus there is no feedback about whether the external pipeline succeeded / failed in the first pipeline.
Currently we need to use custom scripts implementing a loop with API polling, which is not a great solution, especially when we want to create a little more complex multi-project pipeline with upstream feedback.
Further details
Use Cases
There are use cases where you want a pipeline to wait for a sub-pipeline to finish, but other cases where you don't. And then even if you waited for it to finish, some cases where you wanted the parent pipeline to fail if the sub-pipeline failed, and other cases where you don't.
expand for usecases
https://gitlab.com/gitlab-org/gitlab-ee/issues/933#note_89155827
From:Failure attribution, mentioned in "MVP" above, is a pretty critical feature. It should really have gone hand-in-hand with triggered builds. From what I remember, Bamboo has supported triggered builds, and more importantly, evaluating the success of triggered builds, for years.
I'm specifically interested in failure attribution for gitlab-ee#39640. I even wrote a small tool called pipeline-commander to trigger the build from within a pipeline. That seems to be far more complicated than letting GitLab's job executor handle it though.
Ideally, we would have a 1st-class YAML field for triggered builds from a job, accepting
some kind of authorization token (mandatory)
project number or namespace-path encoding (mandatory)
git reference, e.g. tag, branch, commit (optional, default to latest from master) In all of those cases it should be possible to use environment variables for filling in values.
Also, to be very clear, the absolute best feature that this would enable is automatic tagging of (candidate?) releases after all testing (including integration) had passed successfully!
That's kind of the Holy Grail of DevOps, no?
https://gitlab.com/gitlab-org/gitlab-ce/issues/39640#note_83420692
From:
- execution of the current job should be suspended, allowing the current runner to execute other jobs
- the triggered pipeline begins execution using any available runner (not necessarily the one that was suspended)
- once the triggered pipeline has finished running, another GitLab runner resumes execution of the suspended job
- if all jobs in the triggered pipeline were successful, then the pipeline status should report success
- otherwise, the triggered pipeline status should report failure (as with any POSIX process, the return value of the process should be non-zero - it would be incredibly useful for GitLab to report that as well, potentially on a per-job basis)
Solution
Implement first-class multi-project pipelines triggers with upstream feedback.
- In https://gitlab.com/gitlab-org/gitlab-ce/issues/52187, a new
trigger:
keyword is added. This builds on that feature by adding an optional strategy value which allows control over the behavior of the trigger:-
strategy: wait
will start a pipeline, wait for it, but proceed regardless of status- The trigger job when having the wait strategy will represent the downstream pipeline's overall status until it finishes. Afterward, regardless of the downstream pipeline status, its status will change to
success
- The trigger job when having the wait strategy will represent the downstream pipeline's overall status until it finishes. Afterward, regardless of the downstream pipeline status, its status will change to
-
strategy: depend
will start a pipeline, wait for it, and fail if the pipeline fails- The trigger job when having the depend strategy will represent the downstream pipeline's overall status until it finishes. Afterward, it will either represent
success
orfailure
depending on the status of the downstream pipeline status
- The trigger job when having the depend strategy will represent the downstream pipeline's overall status until it finishes. Afterward, it will either represent
- If no strategy is set, it will start the pipeline, not wait for it, and never act on the status (existing behavior)
-
- It may currently happen similar to https://gitlab.com/gitlab-org/gitlab-ce/issues/52187 that a downstream pipeline may not be directly tied (visually) to the trigger job. This is existing UX debt and will be fixed upon in a separate issue. The functionality is though untouched by this and works as it should be.
TODO: create issuehttps://gitlab.com/gitlab-org/gitlab-ce/issues/54854 Additional notes: -
TODO:
Uponfailure
, we need a dedicated failure state similar toscript failure
=> Let's go fordownstream failure
=> example screenshot
That won't cover all the cases, but it would cover the majority of them.
Examples:
job:
trigger:
project: my/project
strategy: wait
job:
trigger:
project: my/project
strategy: depend
What does success look like, and how can we measure that?
Upstream pipelines are now able to wait/depend on downstream pipelines to finish and change their status accordingly.