runsv not running

ZD: https://gitlab.zendesk.com/agent/tickets/31286

Description

After restarting the application GitLb doesn't start anymore due to runsv not running . Customer reinstalled with the process described below but still experiences the same result.

Env

GitLab: gitlab-ee_8.10.0-ee.1_amd64.deb
Issue happens after upgrade.
VM:
- 4 cores
- 3.5GHz Xeon
- 2CPU perf cap
- 2GB RAM
- 1GB swap
- Ubuntu 16.04.1 LTS

Reproducing and Logs

lsb_release -rd 
Description: Ubuntu 16.04.1 LTS 
Release: 16.04

uname -rm 
4.4.0-34-generic x86_64

Stop GitLab

gitlab-ctl stop

Doesn't always work, so

gitlab-ctl kill

Clean

gitlab-ctl cleanse

Uninstall

gitlab-ctl uninstall 
... 
This will delete *all* configuration, log, and variable data associated with this application.

*** You have 60 seconds to hit CTRL-C ***

Check

find /opt/gitlab /var/opt/gitlab /usr/lib/systemd/system/gitlab* -type f | wc 
57958 57965 4997848

Remove what uninstall failed to

apt-get -y remove gitlab-ee 
apt-get -y purge gitlab-ee

Check again

find /opt/gitlab /var/opt/gitlab /usr/lib/systemd/system/gitlab* -type f | wc 
248 248 17391

Do it manually

rm -rf \ 
/opt/gitlab \ 
/var/opt/gitlab \ 
/usr/lib/systemd/system/gitlab* \ 
/root/gitlab-cleanse*

Check again

find /opt/gitlab /var/opt/gitlab /usr/lib/systemd/system/gitlab* -type f | wc 
find: ‘/opt/gitlab’: No such file or directory 
find: ‘/var/opt/gitlab*’: No such file or directory 
find: ‘/usr/lib/systemd/system/gitlab*’: No such file or directory 
0 0 0

check further

updatedb 
locate -i gitlab 
/etc/apt/sources.list.d/gitlab_gitlab-ee.list 
/etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf 
/etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf 
/etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf 
/etc/sysctl.d/90-omnibus-gitlab-net.core.somaxconn.conf 
/etc/systemd/system/basic.target.wants/gitlab-runsvdir.service 
/var/cache/apt/archives/gitlab-ee_8.10.5-ee.0_amd64.deb 
/var/lib/apt/lists/packages.gitlab.com_gitlab_gitlab-ee_ubuntu_dists_xenial_InRelease 
/var/lib/apt/lists/packages.gitlab.com_gitlab_gitlab-ee_ubuntu_dists_xenial_main_binary-amd64_Packages

Remove

rm -f $( locate -i gitlab ) 
updatedb 
locate -i gitlab 
(empty)

Tidy up

apt-get autoclean 
apt-get clean 
systemctl daemon-reload

Just to be sure

locate -i gitlab 
(empty) 
ps ax | grep git 
(empty)

Now, start over

Prep

edit /etc/sysctl.conf 
+	vm.overcommit_memory = 1 
sysctl -p

edit /etc/rc.local 
+	echo never > /sys/kernel/mm/transparent_hugepage/enabled 
exit 0 
echo never > /sys/kernel/mm/transparent_hugepage/enabled

Add repo

curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | bash

Precheck

cd ~ 
dpkg --configure -a 
apt-get update -o Acquire::CompressionTypes::Order::=gz 
apt-get upgrade

Install

time apt-get install gitlab-ee 
... 
It looks like GitLab has not been configured yet; skipping the upgrade script.

real 22m10.659s 
user 0m39.752s 
sys 1m42.056s

Sitting at

Unpacking gitlab-ee (8.10.5-ee.0) ...

Configure

gitlab-ctl reconfigure 
... 
Running handlers complete 
Chef Client finished, 205/308 resources updated in 02 minutes 07 seconds 
gitlab Reconfigured!

Checking

ps ax | grep git 
14350 ? Ss 0:00 runsvdir -P /opt/gitlab/service log: ........................................................................................................................................................................................................................................................................................................................................................................................................... 
14356 ? S 0:00 svlogd -tt /var/log/gitlab/redis 
14357 ? Ssl 0:00 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0 
14423 ? S 0:00 svlogd -tt /var/log/gitlab/postgresql 
14424 ? Ss 0:00 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data 
14547 ? S 0:00 svlogd -tt /var/log/gitlab/unicorn 
14579 ? S 0:00 svlogd -tt /var/log/gitlab/sidekiq 
14580 ? Ssl 0:14 sidekiq 4.1.4 gitlab-rails [0 of 25 busy] 
14589 ? Ss 0:00 runsv gitlab-workhorse 
14590 ? S 0:00 svlogd -tt /var/log/gitlab/gitlab-workhorse 
14598 ? S 0:00 svlogd -tt /var/log/gitlab/nginx 
14599 ? Ss 0:00 nginx: master process /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx 
14613 ? S 0:00 svlogd -tt /var/log/gitlab/logrotate 
14614 ? Ss 0:00 /bin/sh /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper 
14633 ? Ss 0:00 postgres: gitlab gitlabhq_production [local] idle 
14702 ? Ssl 0:00 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork unix -listenUmask 0 -listenAddr /var/opt/gitlab/gitlab-workhorse/socket -authBackend http://localhost:8080 -authSocket /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket -documentRoot /opt/gitlab/embedded/service/gitlab-rails/public -pprofListenAddr 
14708 ? Sl 0:12 unicorn master -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru 
14754 ? Ss 0:00 postgres: gitlab gitlabhq_production [local] idle 
14760 ? Sl 0:00 unicorn worker[0] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru 
14763 ? Sl 0:00 unicorn worker[1] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru 
14766 ? Sl 0:00 unicorn worker[2] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru 
14789 ? Ss 0:00 /bin/bash /opt/gitlab/embedded/bin/gitlab-unicorn-wrapper 
14886 pts/0 S+ 0:00 grep git

systemctl status gitlab-runsvdir -l 
● gitlab-runsvdir.service - GitLab Runit supervision process 
Loaded: loaded (/usr/lib/systemd/system/gitlab-runsvdir.service; enabled; vendor preset: e 
Drop-In: /etc/systemd/system/gitlab-runsvdir.service.d 
└─requires.conf 
Active: active (running) since Sat 2016-08-13 08:32:30 PDT; 6min ago 
Main PID: 14350 (runsvdir) 
Tasks: 79 
Memory: 668.5M 
CPU: 47.694s 
CGroup: /system.slice/gitlab-runsvdir.service 
├─14350 runsvdir -P /opt/gitlab/service log: ..................................... 
├─14355 runsv redis 
├─14356 svlogd -tt /var/log/gitlab/redis 
├─14357 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0 
├─14422 runsv postgresql 
├─14423 svlogd -tt /var/log/gitlab/postgresql

Navigate to

http://example.com/admin/application_settings

Change password: works

Sign-in: Works --> "Signed in successfully."

shutdown -r now

Check, not running

ps ax | grep git 
(empty)

systemctl status gitlab-runsvdir.service -l 
● gitlab-runsvdir.service - GitLab Runit supervision process 
Loaded: loaded (/usr/lib/systemd/system/gitlab-runsvdir.service; enabled; vendor preset: e 
Drop-In: /etc/systemd/system/gitlab-runsvdir.service.d 
└─requires.conf 
Active: inactive (dead)

Can't start

gitlab-ctl start 
fail: gitlab-workhorse: runsv not running 
fail: logrotate: runsv not running 
fail: nginx: runsv not running 
fail: postgresql: runsv not running 
fail: redis: runsv not running 
fail: sidekiq: runsv not running 
fail: unicorn: runsv not running

ps ax | grep git 
(empty)

systemctl restart gitlab-runsvdir.service 
systemctl status gitlab-runsvdir.service -l 
● gitlab-runsvdir.service - GitLab Runit supervision process 
Loaded: loaded (/usr/lib/systemd/system/gitlab-runsvdir.service; enabled; vendor preset: e 
Drop-In: /etc/systemd/system/gitlab-runsvdir.service.d 
└─requires.conf 
Active: active (running) since Sat 2016-08-13 09:19:17 PDT; 3s ago 
Main PID: 3543 (runsvdir) 
Tasks: 46 
Memory: 151.5M 
CPU: 2.428s 
CGroup: /system.slice/gitlab-runsvdir.service 
├─3543 runsvdir -P /opt/gitlab/service log: ...................................... 
├─3545 runsv postgresql

Related issues

#1156

/cc @stanhu @marin

Edited Aug 14, 2020 by 🤖 GitLab Bot 🤖