### Problem to solve It's a common scenario that you are developing code that's dependent on a library or other functionality that comes from another project. Whenever that project is rebuilt, you want your pipeline to run in order to handle the latest changes. The only way to do that in GitLab now is to add `trigger` keywords to the project you're depending on. This is less than ideal, however, because that project may not be aware of you, and/or may not want to manage a list of downstream projects that are dependent on it. Context from product scaling agenda: https://docs.google.com/document/d/1nMJzrDfG7C14WP5v7P226oPFuXkwqIk7bdIT8ai0DNU/edit?ts=5d84fb07&skip_itp2_check=true&pli=1#bookmark=id.5oasolagf8br ### Target audience - Sasha, Software Developer, https://design.gitlab.com/research/personas#persona-sasha - Devon, DevOps Engineer, https://design.gitlab.com/research/personas#persona-devon ### Solution We will introduce a new project-level configuration that allows successfully completed pipelines on the default (`master`) branch of one project to trigger a new pipeline on the default (`master`) branch of the subscribing project. Subscriptions are owned by a single creating user, and the triggered pipelines run as that user as well. Only users who are `maintainer` or `owner` are allowed to create subscriptions. There will be a maximum of 100 subscriptions from or to a single project (i.e., project A can have 100 subscriptions to 100 different projects, and can be subscribed to a maximum of 100 times by 100 other projects.) #### Complex Branch Mappings There are some use cases where users may want more complicated branch mappings between projects. For example, a successfully completed pipeline on `master` in the upstream to trigger a pipeline on `development` in the subscribing project. Due to potential complexity in managing this mapping, we are not addressing this use case in the MVC. It's something we may consider for future expansion. #### UI You'll be able to subscribe to an upstream project only from the downstream project's settings. It will be located under CI/CD settings below `pipeline triggers`.  - The dropdown will show projects the current user has enough permissions to. - Link within help text points to the default branch of the current project (if this proves to be troublesome, let's link to default branch documentation). In terms of visualizing the connections inside of the pipeline graphs, my suggestion is there similar to prior comments in this issue to scope this out and not visualizing it at all except for the upstream connection. As there can only be 1 upstream connection we do not have the problem thereof over visualization and it allows to track the origin of the trigger at least for the downstream pipeline. This will make use of already implemented UI.  Note: For CE this UI will be invisible/unaccessible. ##### Edit towards concept - The dropdown input method has serious downsides in terms of responsiveness and added BE scope. - Let's create a new issue for `Dynamic dropdown menu loading` for use in the future. - Inputting the path manually is, for now, the feasible option we will go for as mentioned in https://gitlab.com/gitlab-org/gitlab/merge_requests/18943#note_235162837 - We will __not__ support private projects, thus only public projects as including those as well will mean having additional scope introduced of having to hide private project information such as Name and URL information. - We do require validation of the inputted path if it is not supported or found because of this input method. - We need to make sure we do not allow for reverse engineering project paths => thus needing to generalise on error message - Validation will happen on form submission instead of live validation as the user types (this would otherwise introduce additional scope). - Copy to go for in case the project path is not found, has a typo, or is unsupported: `This project path either does not exist or is private` For the design, this will mean the following:   <details> <summary>Rejected `.gitlab-ci.yml`-based Solution</summary> #### Rejected `.gitlab-ci.yml`-based Solution We had previous discussed introducing a new `needs:` keyword in the `.gitlab-ci.yml`, but this proved unworkable for a few reasons: * What happens with all the copies of `.gitlab-ci.yml` in different branches? Do each of them also get triggered based on what they point to? The answer feels like, in most cases, these should not trigger pipelines (i.e., if you have 200 branches, you don't want 200 pipelines triggered every time your upstream finishes.) * Similar questions with forks, except the answer to if you want it triggered is probably yes.. most of the time. * Related to the above two, what limitations should there be on triggering and where should they be enforced? We probably don't want there to be thousands of subscriptions on a single project, but if all branches and forks are subscribed this could quickly happen, creating a foot gun. If you edit the `.gitlab-ci.yml` when there are too many subscriptions, do we reject the commit? That feels weird. * What user does the pipeline run as in the triggered pipeline? It has to be *some* user. It probably shouldn't be whoever happened to check in the `.gitlab-ci.yml` last. It also probably shouldn't be the user from the upstream because there may not be any connection between users of one group and the other </details> <details> <summary>Original Solution</summary> Instead of defining the relationship in the source project, we will introduce a new top-level keyword that allows a `gitlab-ci.yml` to specify that it `needs` an upstream, and whenever a pipeline completes in that other project (on the optionally specified branch), it will trigger itself. ```yaml needs: pipeline: project: gitlab-org/gitlab-ce ``` This example would specify that the `.gitlab-ci.yml` in question should be triggered when a pipeline in the `gitlab-org/gitlab-ce` project completes on the branch `master`. From the upstream project's perspective, there's no connection with the downstream (i.e., a job is not added to any upstream pipeline to connect the two as in this [example](https://gitlab.com/gitlab-org/gitlab-ee/uploads/d53025fd4e434c5990a5578bd07482ed/image.png)). The triggering pipeline will be connected to the triggered one, however, for context. In order for one project to track another project, it must have basic read access to the upstream project. This should also shape how we represent these relations in the UI. This means that a subscription is allowed if: 1. A public project subscribes to a public project 1. A private project subscribes to a public project 1. A private project subscribes to a private project (if the user that the downstream pipeline impersonates has read access to the upstream project) ### Implementation Steps There are two backstage issues that need to be completed before this can be finalized: - https://gitlab.com/gitlab-org/gitlab-ee/issues/12334 (~backstage) which implements `needs:pipeline` - https://gitlab.com/gitlab-org/gitlab-ee/issues/11827 (~backstage) which tracks dependencies for automatically triggering pipelines - This issue itself, which covers implementing a service and a worker that triggers downstream dependent pipelines and removes them from the tracking table. ### Behavior As far as how upstream and downstream pipelines are connected to each other: - Upstream / downstream configuration will be persisted after running a pipeline with the upstream trigger on the default branch of the downstream project - Downstream pipelines will be triggered using a bridge job (definition is already in the issue description) - Allows us to reuse existing UI [example](https://gitlab.com/brendan-demo/microservices/b-downstream-1/pipelines/47011467) - Allows us to build on existing job functionality - Other jobs in the same stage as the bridge job will be run as normal This will work similarly to https://gitlab.com/gitlab-org/gitlab-ce/issues/52187 as far as how the token works (the current API expects you to have pre-generated tokens for the downstream project, but as a user, I'd like to be able to just specify the project name and let some other permission system allow the transaction.) Visually, there are a few considerations: - Do not show downstreams in upstream pipelines which are triggered by `needs:pipeline` [example of upstream pipeline](/uploads/d53025fd4e434c5990a5578bd07482ed/image.png) - Do show upstreams in downstream pipelines that use `needs:pipeline` (nothing changes, see an [example pipeline](https://gitlab.com/brendan-demo/microservices/b-downstream-1/pipelines/47011467)) - Otherwise, rely on functionality introduced with https://gitlab.com/gitlab-org/gitlab-ee/issues/8997 * For this issue the trigger job (created by `needs:pipeline`) will just succeed after the pipeline has started * Trigger jobs will have the tag/badge `trigger` * [triggered by api jobs](https://docs.gitlab.com/ee/ci/triggers/) will have its tag changed to `triggered by api` In terms of the edge cases discussed at https://gitlab.com/gitlab-org/gitlab-ee/issues/9045#note_141007066. They are either tackled with the points above or they should not be tended to in this iteration (we just rely on functionality introduced with https://gitlab.com/gitlab-org/gitlab-ee/issues/8997). Let's do a follow up iteration which nails the edge cases and fixes any UX issues. ### Limitations - Only the configuration in the `master` branch `.gitlab-ci.yml` is considered, and only pipelines for `master` can be triggered. - It is not possible in the MVC to monitor a branch other than `master` in the upstream. </details> ### Documentation <!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html Add all known Documentation Requirements here, per https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements --> ### What does success look like, and how can we measure that? <!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. --> ### What is the type of buyer? (Which leads to: in which enterprise tier should this feature go see https://about.gitlab.com/handbook/product/pricing/#four-tiers ) ### Links / references https://gitlab.com/gitlab-org/gitlab-ce/issues/3743#note_5111288