Upgrade notes for 12.3 should contain help about gitaly and changed ulimit behaviour
(not sure where to place it as it is not strictly a bug)
Upgrade from 12.2.5 CE to 12.3.0 CE on Debian stretch from gitlab's repo can fail if run in environments where gitaly
's new ulimit -n 15000
call does not work.
Error messages during update look like
* ruby_block[wait for gitaly service socket] action run (skipped due to not_if)
================================================================================
Error executing action `run` on resource 'ruby_block[restart_service]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
STDOUT: timeout: down: /opt/gitlab/service/gitaly: 1s, normally up, want up
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1
and
down: gitaly: 1s, normally up, want up; run: log: (pid 321) 3107426s
down: gitaly: 1s, normally up, want up; run: log: (pid 321) 3107427s
down: gitaly: 1s, normally up, want up; run: log: (pid 321) 3107428s
[...]
Solution is to point users to the changed ulimit
setting introduced by
* template[/opt/gitlab/sv/gitaly/run] action create
- update content in file /opt/gitlab/sv/gitaly/run from 95d196 to 708519
--- /opt/gitlab/sv/gitaly/run 2019-04-23 06:10:19.056301356 +0000
+++ /opt/gitlab/sv/gitaly/.chef-run20190923-29618-6mmjkj 2019-09-23 06:07:06.366413246 +0000
@@ -8,6 +8,8 @@
cd /var/opt/gitlab/gitaly
+ulimit -n 15000
+
exec chpst -e /opt/gitlab/etc/gitaly/env -P \
-U git:git \
-u git:git \
* directory[/opt/gitlab/sv/gitaly/log] action create (up to date)
Solution for me (using LXC) is simply to set limits.kernel.nofile = 16384
into the config file for the container.