Fetch pipeline refs by name in CI runner
This reverts af43dd6a with the aim of
reducing Gitaly server resouce usage when fetching Git data for CI
builds. Instead of fetching the pipeline ref by SHA, we go back to
fetching it by name 'refs/pipelines/1234'. This causes git fetch
to
use the 'ref-prefix' option (1) when fetching refs. At the time of
af43dd6a, the server side
implementation of 'ref-prefix' was very inefficient. This got fixed in
Git by (2).
Currently each CI fetch causes a full ref walk on the Gitaly server. By going back to fetching by ref, this changes to a filtered refwalk which is much faster. Especially on repositories with many (server side) refs, such as gitlab-org/gitlab.
If you look at the flamegraphs in gitlab-com/gl-infra/scalability#1584 (closed) you can see how much time can be spent on the server on these reference advertisements. The problem is not that the reference advertisements are slow; they are fast enough for CI. The problem is that they run at a high degree of concurrency and during peak CI fetch load they make up a large part of the Gitaly server workload. It is worth reducing the amount of server-side work per reference advertisement because we do so many of them.