### Description GitLab CI/CD pipelines are pretty powerful. Sequential stages and parallel jobs provide a lot of configurability to handle a wide variety of needs. But sometimes it's not enough. Or at least, sometimes it's not efficient enough. Sometimes you want a job in a future stage to run as soon as the job that it depends on finishes. For example, when a project generates both Android and iOS apps in a multi-stage pipeline, people want the iOS deployment to start as soon as all the iOS tests pass rather than waiting for all the Android tests to pass too. The total compute time might be the same, but the wall-clock time is different. In more complicated cases, it's possible to significantly reduce the overall wall-clock time of the pipeline by declaring exactly which jobs depend on which other jobs. A solution like DAG can allow pipelines to be mapped in terms of dependencies, and then cloud compute resources applied automatically in the most efficient way in order to execute. This is very powerful and solves much manual optimization when it comes to pipelines. ### Proposal Implement a simple extension to CI jobs that will allow multiple stages to run at the same time: ```yaml build: stage: build script: echo Hello World setupenv: stage: build rspec: stage: test needs: [build] ``` This causes the `rspec` job to run immediately after the `build` finishes, regardless of status of other jobs in `stage: build`. ### More complex example Here is an in progress pipeline where you can see that the `test_quickbuild` and `deploy_quickbuild` were allowed to proceed even though `longbuild` was still in progress:   <details> <summary>Here is the complete `.gitlab-ci.yml` for implementing this</summary> ```yaml stages: - build - test - deploy longbuild: stage: build script: - sleep 120 quickbuild: stage: build script: - sleep 10 test_quickbuild: stage: test script: - sleep 10 needs: [quickbuild] test_longbuild: stage: test script: - sleep 10 needs: [longbuild] deploy_quickbuild: stage: deploy script: - sleep 10 needs: [test_quickbuild] deploy_longbuild: stage: deploy script: - sleep 10 needs: [test_longbuild] ``` </details> ### The logic of `needs:` 1. If `needs:` is set to point to a job that is not instantiated because of `only/except` rules or otherwise does not exist, the job will fail. 1. Note that one day one of the launch, we are temporarily limiting the maximum number of jobs that a single job can need in the `needs:` array. Track our [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7541) for details on the current limit. 1. If you use `dependencies:` with `needs:`, it's important that you do not mark a job as having a dependency on something that won't have been run at the time it needs it. It's better to use both keywords in this case so that GitLab handles the ordering appropriately. 1. It is impossible for now to have `needs: []` (empty needs), the job always needs to depend on something, unless this is the job in the first stage (see [gitlab-ce#65504](https://gitlab.com/gitlab-org/gitlab-ce/issues/65504)). 1. If `needs:` refers to a job that is marked as `parallel:`. the current job will depend on all parallel jobs created. 1. `needs:` is similar to `dependencies:` in that needs to use jobs from prior stages, this means that it is impossible to create circular dependencies or depend on jobs in the current stage (see [gitlab-ce#65505](https://gitlab.com/gitlab-org/gitlab-ce/issues/65505)). 1. Related to the above, stages must be explicitly defined for all jobs that have the keyword `needs:` or are referred to by one. ### Needing jobs from the same stage For the MVC, we are limiting the definition of `needs:` to only support needing jobs from previous stages. The reason for this is to avoid the risk of circular dependencies. This may be added in a future release (see https://gitlab.com/gitlab-org/gitlab-ce/issues/65505), but we want to see how the feature is used and performs prior to implementing this as a possibility. ### When to run a job that `needs:` another job The keyword `needs:` implies that the job is needed, and should run. Therefore for the MVC we have implemented it so that if a job points to a job that ends up not existing for whatever reason (likely an `only/except` rule that did not match) then the job that `needs` it will also not run. ### Future improvements - See https://gitlab.com/groups/gitlab-org/-/epics/1716 for planned and potential follow-ups. ### Links / references * Prior issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/41947 * Delayed job issue which would benefit from ``needs`` behavior: #51352 * Cross-project `triggered-by:` MVC: https://gitlab.com/gitlab-org/gitlab-ee/issues/9045 * Cross-project build artifacts dependencies: https://gitlab.com/gitlab-org/gitlab-ce/issues/14728