<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=219371) </details> <!--IssueSummary end--> ## Problem Today we have 2 ways of processing jobs in a pipeline: stage and DAG. To distinguish between the 2 types we use `scheduling_type: :stage|:dag` and based on that we would either consider the status of the previous stages or of the defined DAG `needs:` for a given job. This makes our pipeline processing having 2 strategies and likely 2 divergent behaviors. A good example of why this is annoying is that you can't use DAG visualizations on a complex `stages` based pipeline, even if it would be interesting to see them that way. Instead we will have two tabs for looking at them, and only one of the tabs will work for any given pipeline, which is a bit sad. Over time there will be more and more of these, so lets address this split earlier than later. ### Intended users * [Devon (DevOps Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#devon-devops-engineer) ### User experience goal A UX goal here would be to make it easier for people to start taking advantage of `needs` because their pipelines are already `needs` behind the scenes. They wouldn't need to rethink the architecture or switch into a different mode (either in the product or in their way of thinking). ## Proposal Given that a stage-based pipeline is also a DAG-based pipeline with some syntactic sugar, we should be able to default to using a DAG processing for any pipeline. We would at least need `required:false` as discussed at https://gitlab.com/gitlab-org/gitlab/-/issues/30680#note_346587984. What other elements of the DAG are missing to properly model a stages based pipeline? One way we could model stages is something like this, where stages (when defined) become special entities: ```mermaid graph TD; build1 --> build_stage; build2 --> build_stage; build3 --> build_stage; build4 --> build_stage; build_stage --> test12; build_stage --> test12; build_stage --> test34; build_stage --> test34; test12 --> deploy_stage; test34 --> deploy_stage; deploy_stage --> deploy_all; ``` This could interact with a pure-DAG set of jobs as follows; in the case of build5 it runs completely independently, and in the case of build6 it joins back up for the deployment: ```mermaid graph TD; build5 --> test5; test5 --> deploy5; build6 --> test6; test6 --> deploy_all; build1 --> build_stage; build2 --> build_stage; build3 --> build_stage; build4 --> build_stage; build_stage --> test12; build_stage --> test12; build_stage --> test34; build_stage --> test34; test12 --> deploy_stage; test34 --> deploy_stage; deploy_stage --> deploy_all; ``` One thing we would have to figure out is what to do with stages in a pure DAG, since we still allow them to be defined but they become something more like labels. The most difficult part I think will be making it so that existing `.gitlab-ci.yml`s that don't use `needs` can be interpreted implicitly as having `needs` without them having to do anything. TBD - introduce minimal changes ensuring that: - no changes are made to the YAML syntax - no changes are made to the pipeline behavior /cc @ayufan ### Further details <!-- Include use cases, benefits, goals, or any other details that will help us understand the problem better. --> ### Permissions and Security <!-- What permissions are required to perform the described actions? Are they consistent with the existing permissions as documented for users, groups, and projects as appropriate? Is the proposed behavior consistent between the UI, API, and other access methods (e.g. email replies)?--> ### Documentation <!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/workflow.html#for-a-product-change * Add all known Documentation Requirements in this section. See https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements * If this feature requires changing permissions, update the permissions document. See https://docs.gitlab.com/ee/user/permissions.html --> ### Availability & Testing <!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier. What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance. * Unit test changes * Integration test changes * End-to-end test change See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning --> ### What does success look like, and how can we measure that? <!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. --> ### What is the type of buyer? <!-- What is the buyer persona for this feature? See https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/buyer-persona/ In which enterprise tier should this feature go? See https://about.gitlab.com/handbook/product/pricing/#four-tiers --> ### Is this a cross-stage feature? <!-- Communicate if this change will affect multiple Stage Groups or product areas. We recommend always start with the assumption that a feature request will have an impact into another Group. Loop in the most relevant PM and Product Designer from that Group to provide strategic support to help align the Group's broader plan and vision, as well as to avoid UX and technical debt. https://about.gitlab.com/handbook/product/#cross-stage-features --> ### Links / references