Investigate putting gitaly into memory limited cgroups

This issue is to track corrective action for https://gitlab.com/gitlab-com/infrastructure/issues/2314

Simplest boring solution would be something like that:

  1. prepare cgroup mkdir /sys/fs/cgroup/memory/gitaly
  2. limit memory for this cgroup echo 50G > /sys/fs/cgroup/memory/gitaly/memory.limit_in_bytes
  3. disable swap for this cgroup echo 0 > /sys/fs/cgroup/memory/gitaly/memory.swappiness

The above can be done manually of put into rc.local by chef to run on every boot.

After that, we can put all gitaly processes and their children inside that cgroup by simply doing ps aux | grep [g]italy | awk '{print $2}' | xargs -n1 -I{} sh -c 'echo {} > /sys/fs/cgroup/memory/gitaly/cgroup.procs', or hack the startup script.

As soon as collective memory usage hits the limit (50G in this case), the OOM will start killing processes inside that cgroup w/o impacting the rest of the system. The numbers are subject to discussion: we have 55G RAM on those boxes of which 40 or so is fs cache.

Of course, this is the simplest approach, maybe containerizing the application in the long run is better.

/cc @gl-infra @andrewn