Skip to content
Snippets Groups Projects

Switch Git distribution to zlib-ng

    • Closed Issue created by Patrick Steinhardt @pks-gitlab

    The zlib-ng project is a hard fork of the zlib project. The intent of this project is to be a bit less conservative than zlib itself. Most importantly, the project has implemented several optimizations, like using AVX2 when available. This leads to a significant speedup compared to zlib in several cases.

    Git depends on zlib for its object database: every object is stored compressed with zlib. As such, every single object read needs to deflate data. It follows that a speedup for zlib would thus directly translate into a speedup for object lookups, which should in turn speed up almost every single RPC in Gitaly. A benchmark shows that we can achieve up to 25% speedups like this:

    Benchmark 1: /home/pks/Development/git/build-default/git cat-file --batch --batch-all-objects
  Time (mean ± σ):     53.079 s ±  0.512 s    [User: 52.472 s, System: 0.467 s]
  Range (min … max):   52.719 s … 53.665 s    3 runs

Benchmark 2: /home/pks/Development/git/build-zlib-ng/git cat-file --batch --batch-all-objects
  Time (mean ± σ):     40.574 s ±  0.156 s    [User: 40.026 s, System: 0.439 s]
  Range (min … max):   40.413 s … 40.725 s    3 runs

Summary
  /home/pks/Development/git/build-zlib-ng/git cat-file --batch --batch-all-objects ran
    1.31 ± 0.01 times faster than /home/pks/Development/git/build-default/git cat-file --batch --batch-all-objects

    This would be a huge performance improvement as it applies globally across almost everything that Gitaly does.

    zlib-ng is also drop-in compatible with zlib, so there are no code changes required. We do have to update our build infra though to make Git link against zlib-ng instead of linking against zlib.

    Edited by Patrick Steinhardt

    Designs

      Child items ...

      2. Activity

      Please register or sign in to reply