## Status - Production rollout being discussed in: https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/344#note_1075709127 The stages for the rollout are: 1. [x] Add the observability required to monitor and debug 2. [x] Release on staging 3. [x] Gather the data required to set appropriate limits for the production rollout 5. [x] Percantage-based rollout in production 5. [x] 100% rollout in production 5. [x] Monitor production for an extra 4 days 6. [x] Documentation 7. [x] ~~Operation improvements~~ Moved to https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/831 Additional notes: - Steve is on-call 6th of Sept - Matt is on-call 13th of Sept - Steve and Matt are OOO for a few weeks at the end of Sept Updates: - 2022-09-01: https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/344#note_1085437103 - 2022-09-08: https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/344#note_1092823289 - 2022-09-16: https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/344#note_1103068543 ## Overview In https://gitlab.com/gitlab-org/gitaly/-/issues/3049 we started the initial discussion to creating [`cgroups`](https://man7.org/linux/man-pages/man7/cgroups.7.html) on a per repo basis, so each repository can only use an allocated CPU and Memory. With this has been a lot of work done from the Gitaly team: - [Initial RFC](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2604) - [PoC](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/11647) - [Documentaiton](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/cgroups.md) Some general facts: - Using https://github.com/containerd/cgroups. - We are using only v1. - Limiting CPU and Memory usage on a repository basis ## Exit criteria - `cgroups` is rolled out to production and is enforcing reasonable limits. - We have enough documentation/runbooks for the on-call to debug the problem. - Observability around cgroups. ## Results We wanted to do this project because we recognized it as a broad mitigation for a wide range of current and future saturation risks. This implementation of cgroups prevents any one project from starving other projects. There are several known ways (and certainly also some undiscovered ways) to induce unbounded memory and CPU usage on Gitaly nodes. In general, an unbounded resource usage pattern can only be fully prevented by adding an upper bound. Efficiency improvements can help reduce incident frequency, but they cannot prevent incidents. For example, if we somehow made git object traversal 5x more memory efficient, that helps the general case, but a bad actor can still trigger the same saturation behavior by adding 5x more objects to their repo or running 5x more concurrent commands. Cgroups provides that missing bounding behavior. In designing a resource isolation model, we chose the customer-oriented boundary of per-project limits. This boundary is easy for users to understand and work with, and it matches well with most of the saturation incidents observed in production, where a single project's git commands collectively saturated the host's CPU or memory. We calibrated the limits to ensure that the normal workload on all Gitaly nodes would not approach the limits. All projects still have a generous burst capacity, but that ceiling is now less than the machine's full capacity. Enough capacity is now reserved that all other projects on the Gitaly host can continue with their normal workload while any one project is bursting to its limit of CPU or memory usage. During and after the rollout, we confirmed that the normal workload is not reaching the limits. This new isolation property prevents several flavors of saturation incident, such as the ones described in the initial proposal. At the time of writing (2022-10-27) we have prevented 5 incidents. In the past, this would have resulted in an apdex dip on the Gitaly nodes, paged the on-call and it wasn't actionable. More detail in https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/344#note_1148660009 Another perspective worth highlighting is the trend in alert volume for the gitaly service over the last 6 months:  https://nonprod-log.gitlab.net/goto/a60c4920-560d-11ed-9af2-6131f0ee4ce6 There may be some other factors at play, but this certainly seems to have had a positive impact.