Investigate Gitaly's behaviour with cgroups running in Kubernetes
Current state of Cgroups in Gitaly
Gitaly provides the ability to configure and run Git processes within cgroups. It offers repository-level isolation, allowing for individual CPU and memory limits to be set for each repository. While there are root level limits, the cgroup hierarchy is structured hierarchically at the repository level.
As of Kubernetes v1.25, cgroups v2 has been a stable release. It comes with requirements on how to run it. The requirements detail specific linux distributions and kernel versions to be used. It also needs configuration to use the systemd cgroup driver.
Note: Starting with v1.22 and later, when creating a cluster with kubeadm, if the user does not set the
cgroupDriver
field underKubeletConfiguration
, kubeadm defaults it tosystemd
.
Gitaly's official docs on how to work with cgroups: Gitaly Docs on Cgroups.
Cgroups in Kubernetes
The kubelet and the underlying container runtime need to interface with cgroups to enforce resource management for pods and containers which includes cpu/memory requests and limits for containerized workloads.
Work to be done:
- Determine the best way to utilise Gitaly with cgroups and recommend the appropriate cgroup hierarchy that runs well in Kubernetes
- Verify Gitaly is running well with respect to cgroups v2's requirements
- Investigate need to migrate cgroups v1 to cgroups v2 in kubernetes
- Stretch: A new feature came out in v1.27 that allows resources to be resized in place. This may be helpful during start up or during peak times. Investigate this dynamic scaling pod limits without restarting and the applicability of Kube Startup CPU Boost, a K8s operator to help with startup.
Reference: Kubernetes Cgroups