Git fsck memory leak

Summary

tl;dr

Git 2.14.3 has a memory leak in git fsck. GitLab uses git fsck for house keeping.

More details

Somewhere in Git 2.14 a memory leak was introduced in the fsck implementation. The regression has already been identified by the Git team and it was fixed. But versions at least up to Git 2.16.2 contain the bug.

The GitLab Omnibus package for 10.4.5 contains Git 2.14.3 and that version has the memory leak. The effect is that once GitLab starts housekeeping on a repository with git fsck that process will easily allocate several GiB of memory.

External References

Other people on the Git mailing list facing most likely the same problem:
https://marc.info/?l=git&m=151860383915388&w=2

Most likely the commit which fixes the problem:
https://github.com/git/git/commit/ba3a08ca0e

Steps to reproduce

The bug is reproducible with Git 2.14.3 by starting git fsck on a Linux kernel repository. On a machine with 6 GiB of RAM and 4 GiB of swap the git process will be suspended by the OOM killer before finishing its task.

What is the current bug behavior?

The memory leak will slow down GitLab and all other processes because the system will be busy swapping.

What is the expected correct behavior?

The system performance is only slightly affected by GitLab housekeeping (git fsck).

Relevant logs

/opt/gitlab/embedded/bin/git --git-dir=/gitlab/git-data/repositories/freescale/linux-fslc.git fsck with Git 2.14.3 will end up like this:
git-fsck

Details of package version

rpm -qa | grep 'gitlab'

gitlab-ce-10.4.5-ce.0.el7.x86_64

gitlab-rake gitlab:env:info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.3.6p384
Gem Version:    2.6.13
Bundler Version:1.13.7
Rake Version:   12.3.0
Redis Version:  3.2.11
Git Version:    2.14.3
Sidekiq Version:5.0.5
Go Version:     unknown

GitLab information
Version:        10.4.5
Revision:       6b6317a
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            http://hidden.com
HTTP Clone URL: http://hidden.com/some-group/some-project.git
SSH Clone URL:  git@hidden.com:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no

GitLab Shell
Version:        5.11.0
Repository storage paths:
- default:      /gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks
Git:            /opt/gitlab/embedded/bin/git

Environment details

  • Operating System: Red Hat Enterprise Linux 7
  • Installation Target, remove incorrect values:
    • VM: sorry, I do not know the details
  • Installation Type, remove incorrect values:
    • Upgrade from version 10.4.3
  • Is there any other software running on the machine:
    • Apache
  • Is this a single or multiple node installation?
    • Single
  • Resources
    • CPU: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
    • Memory total: 6 GiB RAM, 4 GiB swap

Configuration details

I guess this is not relevant. I will provide details if you need them.

Edited by Mark Jonas