## Summary Right now, the `getPipelineDetails` GraphQL query attempts to fetch the entire pipeline in one query. This kind-of works, but there are performance and correctness issues. This is a follow-up of https://gitlab.com/gitlab-org/gitlab/-/issues/469590 to capture the potential proposed follow-up improvements. This issue should probably be lifted into an epic, with one issue per item and more items collected along the way. ## Improvements The query is very large and is probably not sound, because it attempts to fetch the entire pipeline without pagination on the connections. Using an arbitrary pipeline as an example, the largest stage in https://gitlab.com/gitlab-org/gitlab/-/pipelines/1271849272 has 64 "job groups", which is starting to approach the default max page size of 100. If we had a stage with over 100 jobs, then I believe the page would only show the first 100, without any indication that there are more. The same applies for pretty much all connection types. And because the query is both deep _and_ wide, it is extremely slow for large pipelines. Just the response in my example is **1 MB**. A few ideas for improvements: - Fetch only job id as part of needs data. The name is redundant, has a worse upper bound on size than the id, and the same job is often a dependency of many other jobs. ~frontend. Weight 2 - Remove the `needs` data from the initial query and instead fetch it on demand. ~frontend. Weight 3. - Fetch stages first and then the jobs, sorted by `stage_idx` and `created_at` (we have a favourable database index for this). Let the frontend handle job grouping logic instead of offloading it to the GraphQL. ~frontend. Weight 3. - Do something about the `previousStageJobsOrNeeds` field. It makes non-DAG jobs repeat the entire stage before them when just the stage is sufficient. For a two-stage pipeline with $`M`$ and $`N`$ jobs, this results in a lower bound $`\Omega(MN)`$ output size instead of the expected upper bound of $`O(M+N)`$. I think we should deprecate this field in favour of a `previousStageOrJobNeeds`, that returns a `CiStage` for non-DAG jobs, and a collection of jobs otherwise. ~frontend + ~backend. Weight 5. - Offload detailed status display logic to the frontend. The backend should control the logical status, but I don't understand why we're sending icon identifiers and other redundant information all the time. ~frontend + ~backend. Weight 5. ## Risks - The page will probably perform _very slightly_ worse for very small pipelines, because of the extra network requests ## Involved components The pipeline show page. ## Optional: Intended side effects - Pipelines on `gitlab-org/gitlab` should render significantly faster - Enables further ~"feature::enhancement"s - By fetching stages and jobs separately, the resulting queries should be more amenable to pagination - The page can be rendered incrementally as data is loaded in stages, making it feel more responsive ## Optional: Missing test coverage <!-- If you are aware of tests that need to be written or adjusted apart from unit tests for the changed components, please list them here. --> <!-- Please select the appropriate label from the following: ~"feature::addition" ~"type::maintenance" ~"maintenance::refactor" ~"maintenance::pipelines" ~"maintenance::workflow" -->