Unable to install Gitlab CE in LXC container (permission denied on read-only file system)

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

Gitlab CE cannot be installed in LXC container with the default (and secure) settings. The omnibus installer falls on its nose as soon as sysctl values are attempted to be written:

    ================================================================================
    Error executing action `create` on resource 'gitlab_sysctl[kernel.shmmax]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    execute[load sysctl conf kernel.shmmax] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/resources/gitlab_sysctl.rb line 54) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '255'
    ---- Begin output of sysctl -e -p /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf ----
    STDOUT: 
    STDERR: sysctl: setting key "kernel.shmmax": Read-only file system
    ---- End output of sysctl -e -p /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf ----
    Ran sysctl -e -p /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf returned 255

A workaround (found on https://forum.proxmox.com/threads/installing-gitlab-into-lxc-container-sysctl-kernel-shmmax.49388/) is then to adjust the LXC container config and mount procfs as writeable into the container:

lxc.mount.auto = proc:rw

With this workaround the installer continues, but then falls on its nose later again:

Running handlers:
There was an error running gitlab-ctl reconfigure:

execute[reload all sysctl conf] (package::sysctl line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '255'
---- Begin output of sysctl -e --system ----
STDOUT: * Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf ...
kernel.sem = 250 32000 32 262
* Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf ...
kernel.shmall = 4194304
* Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf ...
kernel.shmmax = 17179869184
* Applying /etc/sysctl.d/90-omnibus-gitlab-net.core.somaxconn.conf ...
net.core.somaxconn = 1024
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/protect-links.conf ...
* Applying /etc/sysctl.conf ...
STDERR: sysctl: permission denied on key 'fs.protected_hardlinks'
sysctl: permission denied on key 'fs.protected_symlinks'
---- End output of sysctl -e --system ----
Ran sysctl -e --system returned 255

IMHO, it's not absolutely necessary to set all these Kernel settings. Either the LXC container is created with limits or without and the application installer should merely inform "meh! this setting doesnt look so good" but should not fail the installation. Especially not necessary when the current system values are higher or the same as the ones expected (for example fs.protected_hardlinks was already set to the wanted value).

I only tested this with pure LXC 3.1 on Debian 10 Buster with Kernel 5.10 from backports. LXC container itself was created with Debian Buster as guest as well. I am not sure whether or not this also applies on LXD containers.

Steps to reproduce

What is the current bug behavior?

The installer does not continue as it runs into (sysctl) permission problems.

What is the expected correct behavior?

Issue a warning, that certain Kernel settings could not be applied - but do not fail the installation.

Relevant logs and/or screenshots

see above

Results of GitLab environment info

Gitlab inside a privileged LXC container with Debian 10 LXC Host is running LXC 3.1 on Debian 10 with Kernel 5.10 bpo Gitlab CE 14.1.0-ce.0 from gitlab repositories

Expand for output related to GitLab environment info 
System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.7.2p137
Gem Version:	3.1.4
Bundler Version:2.1.4
Rake Version:	13.0.3
Redis Version:	6.0.14
Git Version:	2.32.0
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	14.1.0
Revision:	b1a082345ce
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.6
URL:		http://gitlab.infiniroot.net
HTTP Clone URL:	http://gitlab.infiniroot.net/some-group/some-project.git
SSH Clone URL:	git@gitlab.infiniroot.net:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	13.19.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Possible fixes

N/A

Edited by 🤖 GitLab Bot 🤖