### Problem to Solve With recently added cool feature of specifying `dependencies` between stages (gitlab-ce#14211) it's now possible to re-use built artifacts across stages by adding a few lines in `.gitlab-ci.yml`. What's missing though is an extension to this feature, allowing to easily specify dependency on some build's some stage's artifact **from another project**. Currently in the build for executable project I'm simply re-building libraries' code every time I build an executable, which is waste of time and resources. The feature of cross-project dependencies would solve the problem elegantly and make life easier, i.e. no need for additional API keys, curl'ing, secure variables, etc. It also may help others requesting features like gitlab-ce#4768 (better links for artifacts). ### Proposal To implement this behavior, we'll add new keywords for `needs` which can reference another project/ref/artifacts in incremental steps as follows: 1. Remove the need for `dependencies` ```yaml generate_content: needs: - job: another-job-in-this-pipeline artifacts: true ``` MR that implements this step: https://gitlab.com/gitlab-org/gitlab/merge_requests/19943 2. Make cross-project job dependency with Bridge ```yaml generate_content: needs: - project: this_project # This is addition to mark this dependency as a Bridge job ref: master # This is to filter for the Bridge job job: another-job-in-another-pipeline artifacts: true ``` This will bring in the last successfully built artifacts from the `another-job-in-another-pipeline` job on the `master` branch of the project `this_project`. ### Reference The following is the original proposal in the issue. > ### Original Example > > > Suppose I have a library project `my-library` and some other project `my-exe` which needs artifacts from `my-library`'s build stage during its build. So, under the proposal, `my-exe`'s `.gitlab-ci.yml` may look something like this: > > ```yaml > ... > > # build job of my-exe project > build:linux: > stage: build > dependencies: > - project: quyse/my-library # mandatory, name of the project > commit: 123456789abcdef123456789abcdef123456789a # optional, commit hash in my-library repository > branch: master # optional, branch name in my-library repository > tag: v1.1 # optional, tag name in my-library repository > # could be other filters > job: linux-x64 # mandatory, job name defined in .gitlab-ci.yml in my-library > script: > ... > > ... > ``` > > So here CI is supposed to fetch artifacts of the `linux-x64` job of the build of `my-library` project made for specified branch and commit (or specified tag), and unpack them into runner's working directory of `my-exe` build (in the same way as it does currently). > > # Notes > > * There's no ambiguity with current usage of `dependencies` field: string value may still mean dependency on some other stage of the same project, and only if it's a YAML map value, then it's an external (cross-project) dependency. > * `commit`, `branch` and `tag` fields are all optional, and work as a filter, they may be used simultaneously to resolve ambiguity: for example there could be multiple builds corresponding to the single commit (in different branches), and of course there're multiple builds corresponding to the single branch. (Actually even with both `commit` and `branch`/`tag` specified it could still be multiple builds, for example, if build has been retried, or branch deleted/re-pushed, or whatever). So if there're multiple builds, CI can just use the latest one. > * Probably CI must always use the latest *successful* build (from the list of builds matching the filter), i.e. the build where all the stages (not marked with `allow_failure`) are successful, not just the one we are using artifacts from. (Just not sure if such a concept already exists). > * If there's no matching builds of the library, the dependent build must wait until one appears (essentially, in the same way as dependent stages are waiting for their dependencies currently). > * CI runner should fetch artifacts using its API token, so it should be allowed to access private projects it has been added to (or if it's added to the group, or whatever other rules apply here). > * It looks reasonable that this feature will only work across projects of the single Gitlab server.