Too many pack-objects processes cause extremely memory usage
Currently running GitLab CE 11.5.4 Omnibus(Gitaly server version is 0.129.0, Git version is 2.18.1) in a corporate environment, our ubuntu vm have 8 core cpu and 64gb memory
As the picture below, many pack-object processes(git pack-objects --revs --thin --stdout --delta-base-offset --include-tag
, during periods of high memory usage, no fetch/clone/pull operating ongoing, so i think the cause of a process is not clone/fetch/pull, but compress some file to a .pack file)cause extremely memory usage and finally whole vm become can't to use
When I use lsof
, I found all processes with high memory usage are the following repository:
ihartcharacter (files 4.3GB with many image resources) ihartscene (files 10.7GB with many image resources) ihartui (files 8.3GB with many image resources)
Since I found this problem, there are more than 500 pack-objects processes concurrence(number of processes operating repo "ihartcharacter" + number of processes operating repo "ihartscene" + number of processes operating repo "ihartui" = all pack-objects processes), some pack-objects processes running very long time(more than 15 hour, because there are a lot of a process, my machine is abnormal after running for more than 15 hours)
Is there any way to solve the high memory usage or limit git pack-objects
processes count? thks