Skip to content

git: Disable alternate refs in fetches to speed them up

Patrick Steinhardt requested to merge pks-git-fetch-disable-alternate-refs into master

We're currently hitting performance issues in repository replication for repositories which have an excessive amount of refs. In the case of "www-gitlab-com", we're talking about 2,1 million refs. This is proving to strain the packfile-negotiation phase for fetches: the client spends all its time marking its own references and then eventually times out.

As part of the packfile-negotiation Git also iterates over references provided by its alternate. And in the case of "www-gitlab-com", 1,8 million refs are "keep-around" refs of the object pool. Using these references as part of the packfile-negotiation is almost sure to not help reduce the size of the packfile though: they refer to objects which are not reachable from the main repository's references anymore. And because we explicitly disable alternate refs when serving a fetch on the remote side, these "keep-around" refs won't play a role in the remote repository either. We're thus wasting time marking these 1,8 million refs.

Disable the use of alternate refs on fetches in the same way we do for git-receive-pack(1). In benchmarks with "www-gitlab-com" this speeds up dry-run fetches by about 20%.

Benchmark 1: mirror-fetch with alternate refs
  Time (mean ± σ):     35.774 s ±  1.335 s    [User: 32.316 s, System: 4.565 s]
  Range (min … max):   34.646 s … 37.248 s    3 runs

Benchmark 2: mirror-fetch without alternate refs
  Time (mean ± σ):     28.264 s ±  0.209 s    [User: 24.956 s, System: 4.076 s]
  Range (min … max):   28.077 s … 28.489 s    3 runs

Summary
  'mirror-fetch without alternate refs' ran
    1.27 ± 0.05 times faster than 'mirror-fetch with alternate refs'

Changelog: performance

Part of gitlab#336657 (closed), #4009 (closed), https://gitlab.com/gitlab-org/gitaly/-/issues/2436

Merge request reports