Trigger pipeline when another project is rebuilt
Problem to solve
It's a common scenario that you are developing code that's dependent on a library or other functionality that comes from another project. Whenever that project is rebuilt, you want your pipeline to run in order to handle the latest changes. The only way to do that in GitLab now is to add
trigger keywords to the project you're depending on. This is less than ideal, however, because that project may not be aware of you, and/or may not want to manage a list of downstream projects that are dependent on it.
Context from product scaling agenda: https://docs.google.com/document/d/1nMJzrDfG7C14WP5v7P226oPFuXkwqIk7bdIT8ai0DNU/edit?ts=5d84fb07&skip_itp2_check=true&pli=1#bookmark=id.5oasolagf8br
- Sasha, Software Developer, https://design.gitlab.com/research/personas#persona-sasha
- Devon, DevOps Engineer, https://design.gitlab.com/research/personas#persona-devon
We will introduce a new project-level configuration that allows successfully completed pipelines on the default (
master) branch of one project to trigger a new pipeline on the default (
master) branch of the subscribing project. Subscriptions are owned by a single creating user, and the triggered pipelines run as that user as well. Only users who are
owner are allowed to create subscriptions.
There will be a maximum of 100 subscriptions from or to a single project (i.e., project A can have 100 subscriptions to 100 different projects, and can be subscribed to a maximum of 100 times by 100 other projects.)
Complex Branch Mappings
There are some use cases where users may want more complicated branch mappings between projects. For example, a successfully completed pipeline on
master in the upstream to trigger a pipeline on
development in the subscribing project. Due to potential complexity in managing this mapping, we are not addressing this use case in the MVC. It's something we may consider for future expansion.
You'll be able to subscribe to an upstream project only from the downstream project's settings. It will be located under CI/CD settings below
- The dropdown will show projects the current user has enough permissions to.
- Link within help text points to the default branch of the current project (if this proves to be troublesome, let's link to default branch documentation).
In terms of visualizing the connections inside of the pipeline graphs, my suggestion is there similar to prior comments in this issue to scope this out and not visualizing it at all except for the upstream connection.
As there can only be 1 upstream connection we do not have the problem thereof over visualization and it allows to track the origin of the trigger at least for the downstream pipeline.
This will make use of already implemented UI.
Note: For CE this UI will be invisible/unaccessible.
Edit towards concept
- The dropdown input method has serious downsides in terms of responsiveness and added BE scope.
- Let's create a new issue for
Dynamic dropdown menu loadingfor use in the future.
- Let's create a new issue for
- Inputting the path manually is, for now, the feasible option we will go for as mentioned in !18943 (comment 235162837)
- We will not support private projects, thus only public projects as including those as well will mean having additional scope introduced of having to hide private project information such as Name and URL information.
- We do require validation of the inputted path if it is not supported or found because of this input method.
- We need to make sure we do not allow for reverse engineering project paths => thus needing to generalise on error message
- Validation will happen on form submission instead of live validation as the user types (this would otherwise introduce additional scope).
- Copy to go for in case the project path is not found, has a typo, or is unsupported:
This project path either does not exist or is private
For the design, this will mean the following:
Rejected `.gitlab-ci.yml`-based Solution
We had previous discussed introducing a new
needs: keyword in the
.gitlab-ci.yml, but this proved unworkable for a few reasons:
- What happens with all the copies of
.gitlab-ci.ymlin different branches? Do each of them also get triggered based on what they point to? The answer feels like, in most cases, these should not trigger pipelines (i.e., if you have 200 branches, you don't want 200 pipelines triggered every time your upstream finishes.)
- Similar questions with forks, except the answer to if you want it triggered is probably yes.. most of the time.
- Related to the above two, what limitations should there be on triggering and where should they be enforced? We probably don't want there to be thousands of subscriptions on a single project, but if all branches and forks are subscribed this could quickly happen, creating a foot gun. If you edit the
.gitlab-ci.ymlwhen there are too many subscriptions, do we reject the commit? That feels weird.
- What user does the pipeline run as in the triggered pipeline? It has to be some user. It probably shouldn't be whoever happened to check in the
.gitlab-ci.ymllast. It also probably shouldn't be the user from the upstream because there may not be any connection between users of one group and the other
Instead of defining the relationship in the source project, we will introduce a new top-level keyword that allows a
gitlab-ci.yml to specify that it
needs an upstream, and whenever a pipeline completes in that other project (on the optionally specified branch), it will trigger itself.
needs: pipeline: project: gitlab-org/gitlab-ce
This example would specify that the
.gitlab-ci.yml in question should be triggered when a pipeline in the
gitlab-org/gitlab-ce project completes on the branch
From the upstream project's perspective, there's no connection with the downstream (i.e., a job is not added to any upstream pipeline to connect the two as in this example). The triggering pipeline will be connected to the triggered one, however, for context.
In order for one project to track another project, it must have basic read access to the upstream project. This should also shape how we represent these relations in the UI.
This means that a subscription is allowed if:
- A public project subscribes to a public project
- A private project subscribes to a public project
- A private project subscribes to a private project (if the user that the downstream pipeline impersonates has read access to the upstream project)
There are two backstage issues that need to be completed before this can be finalized:
https://gitlab.com/gitlab-org/gitlab-ee/issues/12334 (~backstage) which implements
- https://gitlab.com/gitlab-org/gitlab-ee/issues/11827 (~backstage) which tracks dependencies for automatically triggering pipelines
- This issue itself, which covers implementing a service and a worker that triggers downstream dependent pipelines and removes them from the tracking table.
As far as how upstream and downstream pipelines are connected to each other:
- Upstream / downstream configuration will be persisted after running a pipeline with the upstream trigger on the default branch of the downstream project
- Downstream pipelines will be triggered using a bridge job (definition is already in the issue description)
- Allows us to reuse existing UI example
- Allows us to build on existing job functionality
- Other jobs in the same stage as the bridge job will be run as normal
This will work similarly to https://gitlab.com/gitlab-org/gitlab-ce/issues/52187 as far as how the token works (the current API expects you to have pre-generated tokens for the downstream project, but as a user, I'd like to be able to just specify the project name and let some other permission system allow the transaction.)
Visually, there are a few considerations:
- Do not show downstreams in upstream pipelines which are triggered by
needs:pipelineexample of upstream pipeline
- Do show upstreams in downstream pipelines that use
needs:pipeline(nothing changes, see an example pipeline)
- Otherwise, rely on functionality introduced with https://gitlab.com/gitlab-org/gitlab-ee/issues/8997
- For this issue the trigger job (created by
needs:pipeline) will just succeed after the pipeline has started
- Trigger jobs will have the tag/badge
triggered by api jobs will have its tag changed to
triggered by api
- triggered by api jobs will have its tag changed to
- For this issue the trigger job (created by
In terms of the edge cases discussed at https://gitlab.com/gitlab-org/gitlab-ee/issues/9045#note_141007066. They are either tackled with the points above or they should not be tended to in this iteration (we just rely on functionality introduced with https://gitlab.com/gitlab-org/gitlab-ee/issues/8997). Let's do a follow up iteration which nails the edge cases and fixes any UX issues.
- Only the configuration in the
.gitlab-ci.ymlis considered, and only pipelines for
mastercan be triggered.
- It is not possible in the MVC to monitor a branch other than
masterin the upstream.
What does success look like, and how can we measure that?
What is the type of buyer?
(Which leads to: in which enterprise tier should this feature go see https://about.gitlab.com/handbook/product/pricing/#four-tiers )
Links / references