Allow `needs:` (DAG) to refer to a job in the same stage

Release notes

The Needs keyword reduces cycle time, as it ignores stage ordering and runs jobs without waiting for others to complete, which speeds up your pipelines, previously needs could only be created between jobs to different stages (job depends on another job in a different stage), In this release, we've removed this limitation, so you can define a needs relationship between any job you desire, as a result, you can now create a complete CI/CD pipeline without using stages with implicit needs between jobs, so you can define less verbose pipeline which runs even faster.

Problem to solve

With the current implementation of the directed acyclic graph, the user has to help the scheduler a bit by defining stages for jobs, and only passing dependencies between stages. That can get complicated for large DAGs. Right now, users can deal with this by topologically sorting the DAG and greedily adding artificial “stage1”, “stage2”, etc. labels (or even one stage name per job).

Intended users

Individual contributor automators

Further details

Proposal

We will allow to depend on the jobs within the same stage instead of this being prevented by an error.

For now, we are not making stages only a "visualization hint" since they are still part of processing. In the future we are considering making all pipeline processing DAG (just, by default without needs set, it will behave just like a stage-based pipeline). At that point it may make sense to more broadly revisit what stages mean in GitLab CI.

If a job needs another in the same stage, dependencies should be respected and it should wait (within the stage) to run until the job it needs is done. Then, fetch its dependencies and run itself.

Limitations

• Circular references will need to be detected and will still result in an error.
• dependencies: will not be updated to support this (at least as part of this issue) separate from within the context of needs since non-DAG pipelines having dependencies on something in the same stage is undefined.
• We don't yet have a plan to allow needs: to reference items in future stages. This is a more far out there case that could really become difficult to visualize, so we are not addressing it now.

Permissions and Security

Documentation

Testing

What does success look like, and how can we measure that?

Links / references

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.