Allow `needs:` (DAG) to refer to a job in the same stage
Release notes
The Needs keyword reduces cycle time, as it ignores stage ordering and runs jobs without waiting for others to complete, which speeds up your pipelines, previously needs could only be created between jobs to different stages (job depends on another job in a different stage), In this release, we've removed this limitation, so you can define a needs relationship between any job you desire, as a result, you can now create a complete CI/CD pipeline without using stages with implicit needs between jobs, so you can define less verbose pipeline which runs even faster.
Problem to solve
With the current implementation of the directed acyclic graph, the user has to help the scheduler a bit by defining stages for jobs, and only passing dependencies between stages. That can get complicated for large DAGs. Right now, users can deal with this by topologically sorting the DAG and greedily adding artificial “stage1”, “stage2”, etc. labels (or even one stage name per job).
Intended users
Individual contributor automators
Further details
Proposal
We will allow to depend on the jobs within the same stage instead of this being prevented by an error.
For now, we are not making stages only a "visualization hint" since they are still part of processing. In the future we are considering making all pipeline processing DAG (just, by default without needs set, it will behave just like a stage-based pipeline). At that point it may make sense to more broadly revisit what stages mean in GitLab CI.
If a job needs another in the same stage, dependencies should be respected and it should wait (within the stage) to run until the job it needs is done. Then, fetch its dependencies and run itself.
Limitations
- Circular references will need to be detected and will still result in an error.
-
dependencies:will not be updated to support this (at least as part of this issue) separate from within the context ofneedssince non-DAG pipelines having dependencies on something in the same stage is undefined. - We don't yet have a plan to allow
needs:to reference items in future stages. This is a more far out there case that could really become difficult to visualize, so we are not addressing it now.