Skip to content

ci: Reduce persistent ref writes

Stan Hu requested to merge sh-reduce-persistent-ref-writes into master

What does this MR do and why?

CI builds create a persistent Git ref for the pipeline in the form refs/pipelines/:id after each build transitions to the running state. This ref was created to ensure the runner is able fetch to source code because by default Git requires a SHA to be the tip of a persistent Git reference.

Since many CI builds from the same pipeline can transition to the running state, the creation of the this ref can create significant contention on the Git repository. It also causes Gitaly Cluster to add many replication jobs to the queue, which can take over a minute for some large repositories.

We can reduce the amount of ref writes by observing that most of the time we can just create the ref once before the pipeline starts running. If a pipeline finishes, the ref is deleted. However, we may to recreate this ref if a manual build is triggered or a build is retried.

This feature is gated behind ci_reduce_persistent_ref_writes the feature flag.

Note there is currently no way to tell if a build was manually triggered once it transitions out of the manual state. In order to create the ref at the right time, we need to hook into the transition from manual to preparing or pending.

Relates to &7191 (closed)

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Stan Hu

Merge request reports