Upgrade notes for 12.3 should contain help about gitaly and changed ulimit behaviour

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

(not sure where to place it as it is not strictly a bug)

Upgrade from 12.2.5 CE to 12.3.0 CE on Debian stretch from gitlab's repo can fail if run in environments where gitaly's new ulimit -n 15000 call does not work.

Error messages during update look like

* ruby_block[wait for gitaly service socket] action run (skipped due to not_if)
================================================================================
Error executing action `run` on resource 'ruby_block[restart_service]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
STDOUT: timeout: down: /opt/gitlab/service/gitaly: 1s, normally up, want up
STDERR: 
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1

and

down: gitaly: 1s, normally up, want up; run: log: (pid 321) 3107426s
down: gitaly: 1s, normally up, want up; run: log: (pid 321) 3107427s
down: gitaly: 1s, normally up, want up; run: log: (pid 321) 3107428s
[...]

Solution is to point users to the changed ulimit setting introduced by

* template[/opt/gitlab/sv/gitaly/run] action create
- update content in file /opt/gitlab/sv/gitaly/run from 95d196 to 708519
--- /opt/gitlab/sv/gitaly/run     2019-04-23 06:10:19.056301356 +0000
+++ /opt/gitlab/sv/gitaly/.chef-run20190923-29618-6mmjkj  2019-09-23 06:07:06.366413246 +0000
@@ -8,6 +8,8 @@
cd /var/opt/gitlab/gitaly
+ulimit -n 15000
+
exec chpst -e /opt/gitlab/etc/gitaly/env -P \
-U git:git \
-u git:git \
* directory[/opt/gitlab/sv/gitaly/log] action create (up to date)

Solution for me (using LXC) is simply to set limits.kernel.nofile = 16384 into the config file for the container.