Design: Explore alternative pipeline visualizations for very large pipelines
🧩 Problem
The pipeline graph starts to experience performance issues when the pipeline has a very large number of jobs and dependencies. The problems manifest as bugs, reduced speed, and generally poor UX.
On the backend, this can be seen mostly in terms of bug reports on N + 1
-related behavior (#299848 (comment 637216455), #330707 (closed), #323213 (closed), and #335791 (closed)) and other conversations related to job numbers.
On the frontend / in UX, it manifests in terms of an ongoing stream of discussions to decide what to do on an ah-hoc basis (see #330071 for one example).
🎨 Proposal (TBD)
The pipeline graph is used by several different user personas that have different needs, but they all require a smooth, performant experience when viewing the graph.
Sasha mostly uses the pipeline graph (including the job dependencies view) to troubleshoot failed jobs.
Devon and Priyanka use the pipeline graph to troubleshoot failed jobs, generally navigate between the jobs in the pipeline, and to identify and optimize the pipeline critical path.
We should keep these requirements in mind as we explore alternative pipeline visualizations that are more performant for large pipelines.