Skip to content

Skip creation of CI pipeline ref

Stan Hu requested to merge sh-disable-persistent-ref-creation into master

What does this MR do and why?

CI builds create a persistent Git ref for the pipeline in the form refs/pipelines/:id after each build transitions to the running state. This ref was created to ensure the runner is able fetch to source code because by default Git requires a SHA to be the tip of a persistent Git reference.

However, GitLab 13.2 enabled the uploadpack.allowAnySHA1InWant feature by default (gitaly!2349 (merged)), which allows the client to fetch any SHA. As a result, the need to create a persistent ref is no longer needed.

Since many CI builds from the same pipeline can transition to the running state, the creation of the this ref can create significant contention on the Git repository. It also causes Gitaly Cluster to add many replication jobs to the queue, which can take over a minute for some large repositories.

We disable the creation and deletion of this ref with a ci_skip_persistent_ref_create feature flag. We can later refactor or remove the PersistentRef class later if and when we confirm that this ref is no longer needed.

Relates to &7191 (closed)

How to set up and validate locally

  1. Enable the feature:

    Feature.enable(:ci_skip_persistent_ref_create)

  2. Create a project with a CI job, go to Settings -> CI/CD, and disable Auto-cancel redundant pipelines. This is to ensure all new jobs will run.

  3. Create a branch with a README.md and run a script that pushes lots of jobs in a row to the same branch:

while :
do
    echo "hello" >> README.md
    git commit -a -m "test commit"
    git push
done

Check that all jobs are successful.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Stan Hu

Merge request reports