### Summary Test system has 8GB of memory which is the recommended amount: https://gitlab.com/gitlab-org/gitlab-foss/blob/master/doc/install/requirements.md Installation fails due massive memory usage while compiling assets. ### Steps to reproduce I have a vagrant file available to reproduce this problem. You can find it here: https://gitlab.fechner.net/mfechner/Gitlab-vagrant Make sure you have virtualbox, ansible and vagrant installed on the machine you testing it. Execute the following commands (the disc size of FreeBSD virtualbox image must be increased, so you have to execute vagrant up twice the first time): ``` git clone https://gitlab.fechner.net/mfechner/Gitlab-vagrant.git cd Gitlab-vagrant git checkout current vagrant up vagrant destroy cd ~/.vagrant.d/boxes/freebsd-VAGRANTSLASH-FreeBSD-12.1-RELEASE/2019.11.01/virtualbox VBoxManage clonehd vagrant.vmdk tmp-disk.vdi --format vdi VBoxManage modifyhd tmp-disk.vdi --resize 12000 VBoxManage clonehd tmp-disk.vdi resized-disk.vmdk --format vmdk rm -f tmp-disk.vdi vagrant.vmdk mv resized-disk.vmdk vagrant.vmdk cd - vagrant up ``` While the asset compilation is running, you can login with `vagrant ssh` into the machine and use top to see memory consumption. After the memory is full, the kernel starts to kill processes, you can see this in `/var/log/messages`: ``` May 6 08:47:33 freebsd kernel: swap_pager: out of swap space May 6 08:47:33 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:47:35 freebsd kernel: pid 4423 (ruby26), jid 0, uid 211, was killed: out of swap space May 6 08:47:43 freebsd kernel: pid 3789 (postgres), jid 0, uid 770, was killed: out of swap space May 6 08:47:43 freebsd kernel: pid 3791 (postgres), jid 0, uid 770, was killed: out of swap space May 6 08:47:43 freebsd kernel: pid 3786 (postgres), jid 0, uid 770, was killed: out of swap space May 6 08:47:43 freebsd kernel: pid 3788 (postgres), jid 0, uid 770, was killed: out of swap space May 6 08:47:43 freebsd kernel: pid 3787 (postgres), jid 0, uid 770, was killed: out of swap space ``` ### Example Project https://gitlab.fechner.net/mfechner/Gitlab-vagrant.git ### What is the current *bug* behavior? The compilation of assets fails so, gitlab cannot be used. ### What is the expected *correct* behavior? The compilation of assets should run through without any issues, the command that is used for this, matches the documentation here: https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/doc/install/installation.md#compile-assets The command which is used for this is: `su -l git -c "cd /usr/local/www/gitlab-ce && bundle exec rake gitlab:assets:compile RAILS_ENV=production NODE_ENV=production NODE_OPTIONS='--max_old_space_size=4096'"` ### Relevant logs and/or screenshots ``` May 6 08:47:33 freebsd kernel: swap_pager: out of swap space May 6 08:47:33 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:47:35 freebsd kernel: pid 4423 (ruby26), jid 0, uid 211, was killed: out of swap space May 6 08:47:43 freebsd kernel: pid 3789 (postgres), jid 0, uid 770, was killed: out of swap space May 6 08:47:43 freebsd kernel: pid 3791 (postgres), jid 0, uid 770, was killed: out of swap space May 6 08:47:43 freebsd kernel: pid 3786 (postgres), jid 0, uid 770, was killed: out of swap space May 6 08:47:43 freebsd kernel: pid 3788 (postgres), jid 0, uid 770, was killed: out of swap space May 6 08:47:43 freebsd kernel: pid 3787 (postgres), jid 0, uid 770, was killed: out of swap space May 6 08:47:43 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:49:44 freebsd postgres[4966]: [1-1] 2020-05-06 08:49:44.538 UTC [4966] FATAL: lock file "postmaster.pid" already exists May 6 08:49:44 freebsd postgres[4966]: [1-2] 2020-05-06 08:49:44.538 UTC [4966] HINT: Is another postmaster (PID 3784) running in data directory "/var/db/postgres/data11"? May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd syslogd: last message repeated 2 times May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(31): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd syslogd: last message repeated 1 times May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(24): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(24): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(24): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(24): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(24): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(12): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(24): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(24): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(14): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(11): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(7): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(24): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(14): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(11): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(28): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(25): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(19): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(15): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(12): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(5): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(28): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(25): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(22): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(20): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(11): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(25): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(22): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(20): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(11): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(25): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(20): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(3): failed May 6 08:52:42 freebsd syslogd: last message repeated 1 times May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(20): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(3): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(20): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(3): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(20): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(3): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(20): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(3): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(20): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(3): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(20): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(3): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(20): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(3): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(20): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(3): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(20): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(18): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(9): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(19): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(3): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(2): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(7): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd syslogd: last message repeated 1 times May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(12): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(4): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd syslogd: last message repeated 5 times May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(5): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(13): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(32): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(19): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(31): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(21): failed May 6 08:52:42 freebsd kernel: swap_pager_getswapspace(7): failed May 6 08:52:45 freebsd kernel: pid 4970 (ruby26), jid 0, uid 0, was killed: out of swap space May 6 08:52:48 freebsd kernel: swap_pager_getswapspace(32): failed ``` ### Output of checks #### Results of GitLab environment info <details> <summary>Expand for output related to GitLab environment info</summary> <pre> `su -l git -c "cd /usr/local/www/gitlab-ce && bundle exec rake gitlab:env:info RAILS_ENV=production"` System information System: Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 3.0.6 Bundler Version:2.0.2 Rake Version: 12.3.3 Redis Version: 5.0.8 Git Version: 2.26.2 Sidekiq Version:5.2.8 Go Version: unknown GitLab information Version: 12.9.1 Revision: Unknown Directory: /usr/local/www/gitlab-ce DB Adapter: PostgreSQL DB Version: 11.7 URL: http://192.168.0.251:10080 HTTP Clone URL: http://192.168.0.251:10080/some-group/some-project.git SSH Clone URL: ssh://git@192.168.0.251:2223/some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 12.0.0 Repository storage paths: - default: /usr/local/git/repositories GitLab Shell path: /usr/local/share/gitlab-shell Git: /usr/local/bin/git </pre> </details> #### Results of GitLab application Check <details> <summary>Expand for output related to the GitLab application check</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`) (we will only investigate if the tests are passing) </pre> </details> ### Possible fixes