Skip to content

Too many memory allocations in catfile.Batch methods

Here is a 30-second heap allocation profile captured recently on file-43, a Gitaly server that currently serves a lot of FindAllTags and FindLocalBranches traffic.

Screenshot_2021-09-17_at_11.15.44

source data

Note that the total amount is 12GB which is a lot. The biggest chunk (6.3GB) is in PostUploadPack, which we are already working on in gitlab-com/gl-infra&463 (closed). But what I found interesting is that there is so much other excessive heap allocation going on and it mostly has to do with catfile.Batch.

  • 1.4GB of allocations in opentracing (because we create a span for each object we look up?)
  • 2GB of allocations in parsing commits and tags (where we constantly create new bufio buffered readers)

Both of these things are fixable: the spans should probably not be created at this granularity, and parsing buffers can be reused.

Edited by Jacob Vosmaer
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information