## Description Creating a pipeline can sometimes be really slow. Median pipeline creation times for `gitlab-org/gitlab` are [hovering around `40s`](https://log.gprd.gitlab.net/app/r/s/VcI2F) with 95th percentile [around 60s](https://log.gprd.gitlab.net/app/r/s/u3U1G). As pipeline creation initiated via the UI times out after 60s this means that developers internally are often seeing timeouts. This is a lot, and want to improve efficiency and performance. More details in :arrow_right: https://gitlab.com/gitlab-org/gitlab/-/issues/322386 ## Guidelines We will focus primarily on improving the performance of pipeline creation on `gitlab-org/gitlab` project, for the following reasons: - Any improvements to `gitlab-org/gitlab` will cascade to all projects. - Since the observability data we have for the pipeline creation vary a lot depending on the pipeline configuration, project, project configurations, we can focus the attention on one set of data that has less variation. Identify logic we can avoid to execute: - What can we do async? - Example: https://gitlab.com/gitlab-org/gitlab/-/issues/383156+ - What can we defer to later? - Example: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/103377+ Of the things we must execute: - What can we cache? - What can we do in bulk? ## Data After making the changes ensures that improvements are observed in: - `gitlab-org/gitlab` project - https://log.gprd.gitlab.net/goto/5bb29a20-6038-11ed-b0ec-930003e0679c (P99 stacked bar of all creation steps) - all projects - https://log.gprd.gitlab.net/goto/d0a7be00-698e-11ed-9f43-e3784d7fe3ca (P99 stacked bar of all creation steps) **note:** We need to also monitor P50 for the same data above. ## Iterations | Problem | Issue | Status | Notes | | ------- | ----- | ------ | ----- | | Need to reduce the data logged by pipeline creation log as it's impacting infrastructure | https://gitlab.com/gitlab-org/gitlab/-/issues/371636+ | :white_check_mark: | | | Need SSoT metrics we can monitor when making changes | https://gitlab.com/gitlab-org/gitlab/-/issues/383439+ | :white_check_mark: | [dashboard](https://log.gprd.gitlab.net/app/dashboards#/view/5ecae940-6b52-11ed-85ed-e7557b0a598c?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-7d%2Fd%2Cto%3Anow))) | | Assigning encrypted job tokens at creation is very expensive | https://gitlab.com/gitlab-org/gitlab/-/issues/349877+ | :white_check_mark: | | | `Seed` stage spends 50% of the time initializing stub builds for scoped variables | https://gitlab.com/gitlab-org/gitlab/-/issues/350057+ | :octagonal_sign: | Attempted but reverted due to complexity and bug that introduced for little improvement. | | Batching Gitaly requests can improve performance for `include:`-heavy configurations | https://gitlab.com/gitlab-org/gitlab/-/issues/382531+ | :repeat: | Being done by ~"group::pipeline authoring" | | Auto-cancelation of pipelines takes 15-20% of duration and it could be done async | https://gitlab.com/gitlab-org/gitlab/-/issues/383156+ | :repeat: | | | Creation of deployments takes 10% of duration and it could be deferred to later | https://gitlab.com/gitlab-org/gitlab/-/issues/383600+ | :white_check_mark: | | | continue... | | ## Acceptance Criteria We know that not every case can be covered and that especially complex pipelines may take a long time to start. ### FY23Q4 OKR Acceptance Criteria * Decrease average pipeline creation time by 50% * Have tracing data to identify further areas for improvement. ### Full Epic Acceptance Criteria * Pipeline creation in 5 seconds or less p99 for large projects like `gitlab-org/gitlab` * Pipeline creation in 400ms for smaller projects * Light UX treatment to message to users state of creation so they know the pipeline creation is in process/did not fail when in the >5 second case. We can track incremental progress by using Pipeline create on a project that consistently runs the same jobs / pipeline as @stanhu [noted here](https://gitlab.com/groups/gitlab-org/-/epics/7290#note_829697540).