Persist Pipelines sooner to enable auto-merge
Spawned from: #433348 (comment 1680233788)
If we can break down pipeline creation and persist the pipeline sooner, we can set auto-merge on the pipeline before the entire object is done created.
We should have an open discussion about this, as there's a couple of other proposals currently open so maybe a combination of these: #425542
Proposal
- When pipeline creation happens (MR push, manual, schedule, etc) we should handle this in two parts.
- First part: Lightweight creation attributes such as setting the partition_id, checking rate_limits, checking permissions etc
- When this all passes, create and persist a pipeline.
- Then this will call a worker for the 2nd part
- Second part: Everything else
- The async worker will finish creating the pipeline (metrics, populating metadata, cancelRedundant)
Downside: Since pipelines aren't always persisted, we need to order the Ci::CreatePipelineService::SEQUENCES
in a way that all persistance criteria is checked in Part 1.
Goal:
- This reduces the time between pipeline creation & setting auto-merge/MWPS and avoids timeouts as we're shifting the bulk of the process to async
Edited by Max Fan