Reconfigure the Gitaly cgroup cookbook to match new file server nodes

Gitaly currently uses the default cgroup settings (32GB of memory, 1024 CPU shares, etc) from the Gitaly cgroup cookbook.

However, the machines Gitaly is now running on on GitLab.com are much bigger, with much more memory and we should ensure that we're using the optimal settings.

root@file-01-stor-gprd.c.gitlab-production.internal:/home/andrewn# free -g
              total        used        free      shared  buff/cache   available
Mem:            118           2          83           0          31         113
Swap:             0           0           0

113GB of unused memory on file-01

root@file-01-stor-gprd.c.gitlab-production.internal:/home/andrewn# cat /sys/fs/cgroup/memory/system.slice/gitlab-runsvdir.service/memory.max_usage_in_bytes
34359738368

Gitaly cgroup constrained to 32GB of memory, using a maximum of ~20% of available RAM on dedicated Gitaly machines:

^{https://dashboards.gitlab.net/d/Zy6xM95mk/incident-405-failing-api-health-checks?panelId=12&fullscreen&orgId=1&from=now-24h&to=now}

cc @tommy.morgan @jacobvosmaer-gitlab

I think we should take a conservative approach and start off at 58% like we were using in Azure. Azure hosts at 55GB of RAM and we were using 32GB in cgroups for Gitaly.

I propose we use 68G for the new cgroup limit.

https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2395

OK so this looks like we are unnecessarily constraining the OS buffers and cache available to Gitaly -- note the 31 GB in the 'buff/cache' column. Raising that ceiling would be good.

I see the merit of taking a conservative approach but then we must remember to revisit this once NFS is gone. It makes no sense to reserve 49GB of RAM for logging, Chef and Prometheus.

cc @glopezfernandez

On most of the Gitaly nodes, 90% of system memory is totally unused at present....

^{https://prometheus.gprd.gitlab.net/graph?g0.range_input=1d&g0.expr=100%20*%20(1%20-%20((avg_over_time(node_memory_MemFree_bytes%5B1h%5D)%20%2B%20avg_over_time(node_memory_Cached_bytes%5B1h%5D)%20%2B%20avg_over_time(node_memory_Buffers_bytes%5B1h%5D))%20%2F%20avg_over_time(node_memory_MemTotal_bytes%7Btype%3D%22gitaly%22%7D%5B1h%5D)))%0A&g0.tab=0}

@northrup I've commented on your MR: how can we test your change in staging?

changed milestone to %2018-10-06 Completed Milestone

We have been sitting on this issue in awhile so I will put in the next milestone for scheduling. I think what we probably can do is set a node override to test this on a one gitaly server in prod after we test it in staging. If that is ok then role it out. I don't see much risk here, unless I am missing something.

This really is innocuous enough that I don't see why we don't pull this, I have an MR set to go for this, it just needs to be rolled. It's not like we're cutting the memory down, we're expanding what the CGROUP can use and we're moving it so that it still leaves the system with RAM for processing. We've seen time and again that CGROUPS do indeed behave as they were designed.

If nobody objets I'm just going to roll this out like @jarv suggested with one node, observe, and then move on to the rest.

I've configured file-24-stor-gprd.c.gitlab-production.internal with a Chef Node override to move the cgroup from 32GB to 68GB.

assigned to @northrup

I'm going to reconfigure the fleet for 68GB and roll it out one node at a time with delay.

This has been done, the Gitaly fleet is now set to use more RAM.

closed

changed weight to 2

@northrup, looking at various metrics, this change does not appear to have taken effect:

1: cadvisor is reporting 32GB of available memory for the GitLab cgroup

2: free appears to be unchanged from the time when this issue was opened

root@file-02-stor-gprd.c.gitlab-production.internal:/sys/fs/cgroup/memory# free -g
              total        used        free      shared  buff/cache   available
Mem:            118           3          69           1          45         112
Swap:             0           0           0

3: Most of the fleet is still utilizing a small fraction of available memory, see https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4743#note_99469103

I'm going to reopen this while we investigate why it's not rolled out.

reopened

... why didn't that roll out? Now I'm more than a little puzzled. Thanks for reopening this @andrewn.

Well - I found the reason as to why @andrewn ... looks like we made less than awesome decission in the recipe:

https://gitlab.com/gitlab-cookbooks/gitlab_cgroups_gitaly/blob/master/recipes/default.rb#L34-36

So no mater what we set it to after the first run, it'll never be set to anything. I'm deploying the 68GB update and then I'll fix the ridiculous check condition.

From 32 to 68

closed

Nice one @northrup I'm going to dig around and see if it's made any difference to the application.

added ServiceGitaly label