Investigate putting gitaly into memory limited cgroups
This issue is to track corrective action for https://gitlab.com/gitlab-com/infrastructure/issues/2314
Simplest boring solution would be something like that:
- prepare cgroup
mkdir /sys/fs/cgroup/memory/gitaly - limit memory for this cgroup
echo 50G > /sys/fs/cgroup/memory/gitaly/memory.limit_in_bytes - disable swap for this cgroup
echo 0 > /sys/fs/cgroup/memory/gitaly/memory.swappiness
The above can be done manually of put into rc.local by chef to run on every boot.
After that, we can put all gitaly processes and their children inside that cgroup by simply doing ps aux | grep [g]italy | awk '{print $2}' | xargs -n1 -I{} sh -c 'echo {} > /sys/fs/cgroup/memory/gitaly/cgroup.procs', or hack the startup script.
As soon as collective memory usage hits the limit (50G in this case), the OOM will start killing processes inside that cgroup w/o impacting the rest of the system. The numbers are subject to discussion: we have 55G RAM on those boxes of which 40 or so is fs cache.
Of course, this is the simplest approach, maybe containerizing the application in the long run is better.
/cc @gl-infra @andrewn