localrepo: Speed up connectivity check when fetching into pooled repos
In production, we have observed that fetches into biggish pooled repos
have started to time out. The root cause for this is most likely the
connectivity check: when all objects have been retrieved, Git will
verify that we have no missing objects. This connectivity check is
implemented via git rev-list --not --all --alternate-refs --objects
,
where all changed references are written into the stdin of the command.
While the command is already expensive by itself, it gets even more
expensive because it also includes references from alternates. Given
that object pools duplicate all publicly visible references from their
originial pool member and then add several references on top e.g. for
dangling refs, this is growing really expensive.
The following benchmark demonstrates this in gitlab-org/gitlab with parameters taken from a real request that has been canceled in production:
Benchmark #1: git fetch --no-tags target.git cbdb147e293828c4fa2bfc931679d5e2b070053d
Time (mean ± σ): 28.524 s ± 0.501 s [User: 27.219 s, System: 1.983 s]
Range (min … max): 28.170 s … 28.878 s 2 runs
Benchmark #2: git -c core.alternateRefsCommand="# exit 0" fetch --no-tags target.git cbdb147e293828c4fa2bfc931679d5e2b070053d
Time (mean ± σ): 16.390 s ± 0.029 s [User: 15.451 s, System: 0.938 s]
Range (min … max): 16.369 s … 16.411 s 2 runs
Summary
'git -c core.alternateRefsCommand="# exit 0" fetch --no-tags target.git cbdb147e293828c4fa2bfc931679d5e2b070053d' ran
1.74 ± 0.03 times faster than 'git fetch --no-tags target.git cbdb147e293828c4fa2bfc931679d5e2b070053d'
As can be seen, stubbing out alternate references provides a nice speedup of 1.74x.
Disable alternate refs for internal fetches. This really only serves as an initial testbed so we can easily feature-flag the option. If it proves to not surface any problems while providing above speedups, we can globally enable the option for git-fetch(1).
Changelog: performance
Closes: #3231 (closed)