Investigate how to improve Import & Export performance of CI pipelines
<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=542369) </details> <!--IssueSummary end--> It's been observed that CI pipelines Import & Export is significantly slower compared to other types of relations we Export/Import. For example, a project with 7200 pipelines for one of the private projects: 1. Export of CI pipelines only took ~7 hours 1. Import of CI pipelines only took 8 hours This of course depends on the size of the subrelations within the pipeline, but as it currently stands, for a lot of projects, this is extremely inefficient with ~15 pipelines per minute. Here's an example visualization of a project Import from Kibana. As you can see based on logs production the majority of work happens in the first 3h and then the throughput drops significantly. ![image](/uploads/73049758bcaaf3ce9ede164ba5641c9e/image.png){width=483 height=167} ![image](/uploads/d572c1237d57792c4a9e0ed59bc05a47/image.png) We should investigate CI pipelines both Export and Import to see if we can find a way to improve it's performance. We have made [improvements](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/110588) to this relation in the past but it looks like we need to investigate this relation again.
issue