Skip to content

Added multi-pack-index RPC

Please review this together with discussion in #2661 (closed)


Modelled after Microsoft incremental repack algorithm used in https://github.com/microsoft/scalar, we can start adopting git multi-pack-index(midx for short) as a non-intrusive, lightweighted, incremental repack mechanism.

Compared to Gitaly current incremental repack strategy of git -c repack.writeBitmaps=false repack -d, midx repack approach offers options to NOT having to repack everything. With --batch-size=<size>, you can ignore the bigger pack files(often 1) and only consolidate the smaller ones. Note that this repack does not handle loose objects but only packfiles.

I intended this to be used together, not replacing, with current Repack(incremental) RPC. From Gitlab ruby, this should be triggered as agressively as the incremental repack imo.

This should tremendously improve the speed of repacking smaller packs into bigger packs, letting bigger repositories handle multiple packfiles more effectively.

This also opens a pathway to enable receive.unpackLimit=1 config on gitaly, remove the need to decompress transferred-packfiles, received from push activities, into loose objects. This could save Gitlab in term of write count onto slower storage platform(i.e. NFS) and speed up pushes.

This patch was made with reference to Derrick Stolee's proposal to Git core on how to improve git housekeeping activities on client-side.

Edited by Son Luong Ngoc

Merge request reports