Skip to content

Fetch pipeline refs by name in CI runner

Jacob Vosmaer requested to merge jv-ci-refspec into master

This reverts af43dd6a with the aim of reducing Gitaly server resouce usage when fetching Git data for CI builds. Instead of fetching the pipeline ref by SHA, we go back to fetching it by name 'refs/pipelines/1234'. This causes git fetch to use the 'ref-prefix' option (1) when fetching refs. At the time of af43dd6a, the server side implementation of 'ref-prefix' was very inefficient. This got fixed in Git by (2).

Currently each CI fetch causes a full ref walk on the Gitaly server. By going back to fetching by ref, this changes to a filtered refwalk which is much faster. Especially on repositories with many (server side) refs, such as gitlab-org/gitlab.

If you look at the flamegraphs in gitlab-com/gl-infra/scalability#1584 (closed) you can see how much time can be spent on the server on these reference advertisements. The problem is not that the reference advertisements are slow; they are fast enough for CI. The problem is that they run at a high degree of concurrency and during peak CI fetch load they make up a large part of the Gitaly server workload. It is worth reducing the amount of server-side work per reference advertisement because we do so many of them.

Edited by Jacob Vosmaer

Merge request reports

Loading