Skip to content

Gitaly cgroups configuration incompatible with Cgroups V2 ?

(I'm not an expert in any of those subjects though, take this information with a grain of salt)

I tried optimizing my Gitlab docker installation when I found out Gitaly had issues with cgroups.

I used the configuration from gitlab documentation here : https://docs.gitlab.com/omnibus/settings/memory_constrained_envs.html

It proposed to use those parameters to define cpu & memory limits:

gitaly['cgroups_count'] = 2
gitaly['cgroups_mountpoint'] = '/sys/fs/cgroup'
gitaly['cgroups_hierarchy_root'] = 'gitaly'
gitaly['cgroups_memory_enabled'] = true
gitaly['cgroups_memory_limit'] = 500000
gitaly['cgroups_cpu_enabled'] = true
gitaly['cgroups_cpu_shares'] = 512

If I understood right, omnibus generates a gitaly config.toml file, which then contains this parameters :

[cgroups]
count = 2
mountpoint = '/sys/fs/cgroup'
hierarchy_root = 'gitaly'

[cgroups.memory]
enabled = true
limit = 500000

[cgroups.cpu]
enabled = true
shares = 512

Unfortunately, it leads gitaly to try to access a /sys/fs/cgroup/cpu folder, which does not exist because the folder and settings structure of cgroups v2 is different. (see a bit of log from my gitlab docker installation)

==> /var/log/gitlab/gitaly/current <==
mkdir: cannot create directory '/sys/fs/cgroup/cpu': Read-only file system
mkdir: cannot create directory '/sys/fs/cgroup/cpu': Read-only file system
mkdir: cannot create directory '/sys/fs/cgroup/cpu': Read-only file system
mkdir: cannot create directory '/sys/fs/cgroup/cpu': Read-only file system

I also noticed that gitaly cgroup code actually calls for a V1Manager, which, if I understood well, means it doesn't take into account v2 structure at all ? (see a snippet of internal/cgroups/cgroups.go here)

// NewManager returns the appropriate Cgroups manager
func NewManager(cfg cgroups.Config) Manager {
	if cfg.Count > 0 {
		return newV1Manager(cfg)
	}

	return &NoopManager{}
}

It might help for people using cgroups V2 to have a correctly operating set of configuration here to optimize cpu and memory allocation here.

Edited by Mehdi Zakaria Benadel
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information