Skip to content

Trigger pipeline when another project is rebuilt

Problem to solve

It's a common scenario that you are developing code that's dependent on a library or other functionality that comes from another project. Whenever that project is rebuilt, you want your pipeline to run in order to handle the latest changes. The only way to do that in GitLab now is to add trigger keywords to the project you're depending on. This is less than ideal, however, because that project may not be aware of you, and/or may not want to manage a list of downstream projects that are dependent on it.

Context from product scaling agenda: https://docs.google.com/document/d/1nMJzrDfG7C14WP5v7P226oPFuXkwqIk7bdIT8ai0DNU/edit?ts=5d84fb07&skip_itp2_check=true&pli=1#bookmark=id.5oasolagf8br

Target audience

Solution

We will introduce a new project-level configuration that allows successfully completed pipelines on the default (master) branch of one project to trigger a new pipeline on the default (master) branch of the subscribing project. Subscriptions are owned by a single creating user, and the triggered pipelines run as that user as well. Only users who are maintainer or owner are allowed to create subscriptions.

There will be a maximum of 100 subscriptions from or to a single project (i.e., project A can have 100 subscriptions to 100 different projects, and can be subscribed to a maximum of 100 times by 100 other projects.)

Complex Branch Mappings

There are some use cases where users may want more complicated branch mappings between projects. For example, a successfully completed pipeline on master in the upstream to trigger a pipeline on development in the subscribing project. Due to potential complexity in managing this mapping, we are not addressing this use case in the MVC. It's something we may consider for future expansion.

UI

You'll be able to subscribe to an upstream project only from the downstream project's settings. It will be located under CI/CD settings below pipeline triggers.

Settings__1_

  • The dropdown will show projects the current user has enough permissions to.
  • Link within help text points to the default branch of the current project (if this proves to be troublesome, let's link to default branch documentation).

In terms of visualizing the connections inside of the pipeline graphs, my suggestion is there similar to prior comments in this issue to scope this out and not visualizing it at all except for the upstream connection.

As there can only be 1 upstream connection we do not have the problem thereof over visualization and it allows to track the origin of the trigger at least for the downstream pipeline.

This will make use of already implemented UI.

Settings_-Dropdown_menu_copy__1

Note: For CE this UI will be invisible/unaccessible.

Edit towards concept
  • The dropdown input method has serious downsides in terms of responsiveness and added BE scope.
    • Let's create a new issue for Dynamic dropdown menu loading for use in the future.
  • Inputting the path manually is, for now, the feasible option we will go for as mentioned in !18943 (comment 235162837)
    • We will not support private projects, thus only public projects as including those as well will mean having additional scope introduced of having to hide private project information such as Name and URL information.
    • We do require validation of the inputted path if it is not supported or found because of this input method.
      • We need to make sure we do not allow for reverse engineering project paths => thus needing to generalise on error message
      • Validation will happen on form submission instead of live validation as the user types (this would otherwise introduce additional scope).
      • Copy to go for in case the project path is not found, has a typo, or is unsupported: This project path either does not exist or is private

For the design, this will mean the following:

Settings__2_

Validation

Rejected `.gitlab-ci.yml`-based Solution

Rejected .gitlab-ci.yml-based Solution

We had previous discussed introducing a new needs: keyword in the .gitlab-ci.yml, but this proved unworkable for a few reasons:

  • What happens with all the copies of .gitlab-ci.yml in different branches? Do each of them also get triggered based on what they point to? The answer feels like, in most cases, these should not trigger pipelines (i.e., if you have 200 branches, you don't want 200 pipelines triggered every time your upstream finishes.)
  • Similar questions with forks, except the answer to if you want it triggered is probably yes.. most of the time.
  • Related to the above two, what limitations should there be on triggering and where should they be enforced? We probably don't want there to be thousands of subscriptions on a single project, but if all branches and forks are subscribed this could quickly happen, creating a foot gun. If you edit the .gitlab-ci.yml when there are too many subscriptions, do we reject the commit? That feels weird.
  • What user does the pipeline run as in the triggered pipeline? It has to be some user. It probably shouldn't be whoever happened to check in the .gitlab-ci.yml last. It also probably shouldn't be the user from the upstream because there may not be any connection between users of one group and the other
Original Solution

Instead of defining the relationship in the source project, we will introduce a new top-level keyword that allows a gitlab-ci.yml to specify that it needs an upstream, and whenever a pipeline completes in that other project (on the optionally specified branch), it will trigger itself.

needs:
  pipeline: 
    project: gitlab-org/gitlab-ce

This example would specify that the .gitlab-ci.yml in question should be triggered when a pipeline in the gitlab-org/gitlab-ce project completes on the branch master.

From the upstream project's perspective, there's no connection with the downstream (i.e., a job is not added to any upstream pipeline to connect the two as in this example). The triggering pipeline will be connected to the triggered one, however, for context.

In order for one project to track another project, it must have basic read access to the upstream project. This should also shape how we represent these relations in the UI.

This means that a subscription is allowed if:

  1. A public project subscribes to a public project
  2. A private project subscribes to a public project
  3. A private project subscribes to a private project (if the user that the downstream pipeline impersonates has read access to the upstream project)

Implementation Steps

There are two backstage issues that need to be completed before this can be finalized:

Behavior

As far as how upstream and downstream pipelines are connected to each other:

  • Upstream / downstream configuration will be persisted after running a pipeline with the upstream trigger on the default branch of the downstream project
  • Downstream pipelines will be triggered using a bridge job (definition is already in the issue description)
    • Allows us to reuse existing UI example
    • Allows us to build on existing job functionality
    • Other jobs in the same stage as the bridge job will be run as normal

This will work similarly to https://gitlab.com/gitlab-org/gitlab-ce/issues/52187 as far as how the token works (the current API expects you to have pre-generated tokens for the downstream project, but as a user, I'd like to be able to just specify the project name and let some other permission system allow the transaction.)

Visually, there are a few considerations:

In terms of the edge cases discussed at https://gitlab.com/gitlab-org/gitlab-ee/issues/9045#note_141007066. They are either tackled with the points above or they should not be tended to in this iteration (we just rely on functionality introduced with https://gitlab.com/gitlab-org/gitlab-ee/issues/8997). Let's do a follow up iteration which nails the edge cases and fixes any UX issues.

Limitations

  • Only the configuration in the master branch .gitlab-ci.yml is considered, and only pipelines for master can be triggered.
  • It is not possible in the MVC to monitor a branch other than master in the upstream.

Documentation

What does success look like, and how can we measure that?

What is the type of buyer?

(Which leads to: in which enterprise tier should this feature go see https://about.gitlab.com/handbook/product/pricing/#four-tiers )

Links / references

https://gitlab.com/gitlab-org/gitlab-ce/issues/3743#note_5111288

Edited by Jason Yavorska