Gitaly cgroups configuration incompatible with Cgroups V2 ?
(I'm not an expert in any of those subjects though, take this information with a grain of salt)
I tried optimizing my Gitlab docker installation when I found out Gitaly had issues with cgroups.
I used the configuration from gitlab documentation here : https://docs.gitlab.com/omnibus/settings/memory_constrained_envs.html
It proposed to use those parameters to define cpu & memory limits:
gitaly['cgroups_count'] = 2
gitaly['cgroups_mountpoint'] = '/sys/fs/cgroup'
gitaly['cgroups_hierarchy_root'] = 'gitaly'
gitaly['cgroups_memory_enabled'] = true
gitaly['cgroups_memory_limit'] = 500000
gitaly['cgroups_cpu_enabled'] = true
gitaly['cgroups_cpu_shares'] = 512
If I understood right, omnibus generates a gitaly config.toml file, which then contains this parameters :
[cgroups]
count = 2
mountpoint = '/sys/fs/cgroup'
hierarchy_root = 'gitaly'
[cgroups.memory]
enabled = true
limit = 500000
[cgroups.cpu]
enabled = true
shares = 512
Unfortunately, it leads gitaly to try to access a /sys/fs/cgroup/cpu
folder, which does not exist because the folder and settings structure of cgroups v2 is different. (see a bit of log from my gitlab docker installation)
==> /var/log/gitlab/gitaly/current <==
mkdir: cannot create directory '/sys/fs/cgroup/cpu': Read-only file system
mkdir: cannot create directory '/sys/fs/cgroup/cpu': Read-only file system
mkdir: cannot create directory '/sys/fs/cgroup/cpu': Read-only file system
mkdir: cannot create directory '/sys/fs/cgroup/cpu': Read-only file system
I also noticed that gitaly cgroup code actually calls for a V1Manager, which, if I understood well, means it doesn't take into account v2 structure at all ? (see a snippet of internal/cgroups/cgroups.go
here)
// NewManager returns the appropriate Cgroups manager
func NewManager(cfg cgroups.Config) Manager {
if cfg.Count > 0 {
return newV1Manager(cfg)
}
return &NoopManager{}
}
It might help for people using cgroups V2 to have a correctly operating set of configuration here to optimize cpu and memory allocation here.