Added multi-pack-index RPC
Please review this together with discussion in #2661 (closed)
Modelled after Microsoft incremental repack algorithm used in
https://github.com/microsoft/scalar, we can start adopting git multi-pack-index(midx for short) as a non-intrusive, lightweighted,
incremental repack mechanism.
Compared to Gitaly current incremental repack strategy of
git -c repack.writeBitmaps=false repack -d, midx repack approach offers
options to NOT having to repack everything. With --batch-size=<size>,
you can ignore the bigger pack files(often 1) and only consolidate the smaller
ones. Note that this repack does not handle loose objects but only packfiles.
I intended this to be used together, not replacing, with current Repack(incremental) RPC. From Gitlab ruby, this should be triggered as agressively as the incremental repack imo.
This should tremendously improve the speed of repacking smaller packs into bigger packs, letting bigger repositories handle multiple packfiles more effectively.
This also opens a pathway to enable receive.unpackLimit=1 config on gitaly,
remove the need to decompress transferred-packfiles, received from push
activities, into loose objects. This could save Gitlab in term of write
count onto slower storage platform(i.e. NFS) and speed up pushes.
This patch was made with reference to Derrick Stolee's proposal to Git core on how to improve git housekeeping activities on client-side.