Unable to install Gitlab CE in LXC container (permission denied on read-only file system)
Summary
Gitlab CE cannot be installed in LXC container with the default (and secure) settings. The omnibus installer falls on its nose as soon as sysctl values are attempted to be written:
================================================================================
Error executing action `create` on resource 'gitlab_sysctl[kernel.shmmax]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
execute[load sysctl conf kernel.shmmax] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/gitlab_sysctl.rb line 54) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '255'
---- Begin output of sysctl -e -p /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf ----
STDOUT:
STDERR: sysctl: setting key "kernel.shmmax": Read-only file system
---- End output of sysctl -e -p /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf ----
Ran sysctl -e -p /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf returned 255
A workaround (found on https://forum.proxmox.com/threads/installing-gitlab-into-lxc-container-sysctl-kernel-shmmax.49388/) is then to adjust the LXC container config and mount procfs as writeable into the container:
lxc.mount.auto = proc:rw
With this workaround the installer continues, but then falls on its nose later again:
Running handlers:
There was an error running gitlab-ctl reconfigure:
execute[reload all sysctl conf] (package::sysctl line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '255'
---- Begin output of sysctl -e --system ----
STDOUT: * Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf ...
kernel.sem = 250 32000 32 262
* Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf ...
kernel.shmall = 4194304
* Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf ...
kernel.shmmax = 17179869184
* Applying /etc/sysctl.d/90-omnibus-gitlab-net.core.somaxconn.conf ...
net.core.somaxconn = 1024
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/protect-links.conf ...
* Applying /etc/sysctl.conf ...
STDERR: sysctl: permission denied on key 'fs.protected_hardlinks'
sysctl: permission denied on key 'fs.protected_symlinks'
---- End output of sysctl -e --system ----
Ran sysctl -e --system returned 255
IMHO, it's not absolutely necessary to set all these Kernel settings. Either the LXC container is created with limits or without and the application installer should merely inform "meh! this setting doesnt look so good" but should not fail the installation. Especially not necessary when the current system values are higher or the same as the ones expected (for example fs.protected_hardlinks was already set to the wanted value).
I only tested this with pure LXC 3.1 on Debian 10 Buster with Kernel 5.10 from backports. LXC container itself was created with Debian Buster as guest as well. I am not sure whether or not this also applies on LXD containers.
Steps to reproduce
- Create a (privileged) LXC (not LXD!) container with default settings (lxc.apparmor.profile = generated)
- Follow https://about.gitlab.com/install/?version=ce#debian
What is the current bug behavior?
The installer does not continue as it runs into (sysctl) permission problems.
What is the expected correct behavior?
Issue a warning, that certain Kernel settings could not be applied - but do not fail the installation.
Relevant logs and/or screenshots
see above
Results of GitLab environment info
Gitlab inside a privileged LXC container with Debian 10 LXC Host is running LXC 3.1 on Debian 10 with Kernel 5.10 bpo Gitlab CE 14.1.0-ce.0 from gitlab repositories
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.7.2p137 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.3 Redis Version: 6.0.14 Git Version: 2.32.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 14.1.0 Revision: b1a082345ce Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.6 URL: http://gitlab.infiniroot.net HTTP Clone URL: http://gitlab.infiniroot.net/some-group/some-project.git SSH Clone URL: git@gitlab.infiniroot.net:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 13.19.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Possible fixes
N/A