runsv not running
ZD: https://gitlab.zendesk.com/agent/tickets/31286
Description
After restarting the application GitLb doesn't start anymore due to runsv not running
. Customer reinstalled with the process described below but still experiences the same result.
Env
- GitLab:
gitlab-ee_8.10.0-ee.1_amd64.deb
- Issue happens after upgrade.
- VM:
- 4 cores
- 3.5GHz Xeon
- 2CPU perf cap
- 2GB RAM
- 1GB swap
- Ubuntu 16.04.1 LTS
Reproducing and Logs
lsb_release -rd
Description: Ubuntu 16.04.1 LTS
Release: 16.04
uname -rm
4.4.0-34-generic x86_64
Stop GitLab
gitlab-ctl stop
Doesn't always work, so
gitlab-ctl kill
Clean
gitlab-ctl cleanse
Uninstall
gitlab-ctl uninstall
...
This will delete *all* configuration, log, and variable data associated with this application.
*** You have 60 seconds to hit CTRL-C ***
Check
find /opt/gitlab /var/opt/gitlab /usr/lib/systemd/system/gitlab* -type f | wc
57958 57965 4997848
Remove what uninstall failed to
apt-get -y remove gitlab-ee
apt-get -y purge gitlab-ee
Check again
find /opt/gitlab /var/opt/gitlab /usr/lib/systemd/system/gitlab* -type f | wc
248 248 17391
Do it manually
rm -rf \
/opt/gitlab \
/var/opt/gitlab \
/usr/lib/systemd/system/gitlab* \
/root/gitlab-cleanse*
Check again
find /opt/gitlab /var/opt/gitlab /usr/lib/systemd/system/gitlab* -type f | wc
find: ‘/opt/gitlab’: No such file or directory
find: ‘/var/opt/gitlab*’: No such file or directory
find: ‘/usr/lib/systemd/system/gitlab*’: No such file or directory
0 0 0
check further
updatedb
locate -i gitlab
/etc/apt/sources.list.d/gitlab_gitlab-ee.list
/etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf
/etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf
/etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf
/etc/sysctl.d/90-omnibus-gitlab-net.core.somaxconn.conf
/etc/systemd/system/basic.target.wants/gitlab-runsvdir.service
/var/cache/apt/archives/gitlab-ee_8.10.5-ee.0_amd64.deb
/var/lib/apt/lists/packages.gitlab.com_gitlab_gitlab-ee_ubuntu_dists_xenial_InRelease
/var/lib/apt/lists/packages.gitlab.com_gitlab_gitlab-ee_ubuntu_dists_xenial_main_binary-amd64_Packages
Remove
rm -f $( locate -i gitlab )
updatedb
locate -i gitlab
(empty)
Tidy up
apt-get autoclean
apt-get clean
systemctl daemon-reload
Just to be sure
locate -i gitlab
(empty)
ps ax | grep git
(empty)
Now, start over
Prep
edit /etc/sysctl.conf
+ vm.overcommit_memory = 1
sysctl -p
edit /etc/rc.local
+ echo never > /sys/kernel/mm/transparent_hugepage/enabled
exit 0
echo never > /sys/kernel/mm/transparent_hugepage/enabled
Add repo
curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | bash
Precheck
cd ~
dpkg --configure -a
apt-get update -o Acquire::CompressionTypes::Order::=gz
apt-get upgrade
Install
time apt-get install gitlab-ee
...
It looks like GitLab has not been configured yet; skipping the upgrade script.
real 22m10.659s
user 0m39.752s
sys 1m42.056s
Sitting at
Unpacking gitlab-ee (8.10.5-ee.0) ...
Configure
gitlab-ctl reconfigure
...
Running handlers complete
Chef Client finished, 205/308 resources updated in 02 minutes 07 seconds
gitlab Reconfigured!
Checking
ps ax | grep git
14350 ? Ss 0:00 runsvdir -P /opt/gitlab/service log: ...........................................................................................................................................................................................................................................................................................................................................................................................................
14356 ? S 0:00 svlogd -tt /var/log/gitlab/redis
14357 ? Ssl 0:00 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0
14423 ? S 0:00 svlogd -tt /var/log/gitlab/postgresql
14424 ? Ss 0:00 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
14547 ? S 0:00 svlogd -tt /var/log/gitlab/unicorn
14579 ? S 0:00 svlogd -tt /var/log/gitlab/sidekiq
14580 ? Ssl 0:14 sidekiq 4.1.4 gitlab-rails [0 of 25 busy]
14589 ? Ss 0:00 runsv gitlab-workhorse
14590 ? S 0:00 svlogd -tt /var/log/gitlab/gitlab-workhorse
14598 ? S 0:00 svlogd -tt /var/log/gitlab/nginx
14599 ? Ss 0:00 nginx: master process /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx
14613 ? S 0:00 svlogd -tt /var/log/gitlab/logrotate
14614 ? Ss 0:00 /bin/sh /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper
14633 ? Ss 0:00 postgres: gitlab gitlabhq_production [local] idle
14702 ? Ssl 0:00 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork unix -listenUmask 0 -listenAddr /var/opt/gitlab/gitlab-workhorse/socket -authBackend http://localhost:8080 -authSocket /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket -documentRoot /opt/gitlab/embedded/service/gitlab-rails/public -pprofListenAddr
14708 ? Sl 0:12 unicorn master -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru
14754 ? Ss 0:00 postgres: gitlab gitlabhq_production [local] idle
14760 ? Sl 0:00 unicorn worker[0] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru
14763 ? Sl 0:00 unicorn worker[1] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru
14766 ? Sl 0:00 unicorn worker[2] -D -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru
14789 ? Ss 0:00 /bin/bash /opt/gitlab/embedded/bin/gitlab-unicorn-wrapper
14886 pts/0 S+ 0:00 grep git
systemctl status gitlab-runsvdir -l
● gitlab-runsvdir.service - GitLab Runit supervision process
Loaded: loaded (/usr/lib/systemd/system/gitlab-runsvdir.service; enabled; vendor preset: e
Drop-In: /etc/systemd/system/gitlab-runsvdir.service.d
└─requires.conf
Active: active (running) since Sat 2016-08-13 08:32:30 PDT; 6min ago
Main PID: 14350 (runsvdir)
Tasks: 79
Memory: 668.5M
CPU: 47.694s
CGroup: /system.slice/gitlab-runsvdir.service
├─14350 runsvdir -P /opt/gitlab/service log: .....................................
├─14355 runsv redis
├─14356 svlogd -tt /var/log/gitlab/redis
├─14357 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0
├─14422 runsv postgresql
├─14423 svlogd -tt /var/log/gitlab/postgresql
Navigate to
http://example.com/admin/application_settings
Change password: works
Sign-in: Works --> "Signed in successfully."
shutdown -r now
Check, not running
ps ax | grep git
(empty)
systemctl status gitlab-runsvdir.service -l
● gitlab-runsvdir.service - GitLab Runit supervision process
Loaded: loaded (/usr/lib/systemd/system/gitlab-runsvdir.service; enabled; vendor preset: e
Drop-In: /etc/systemd/system/gitlab-runsvdir.service.d
└─requires.conf
Active: inactive (dead)
Can't start
gitlab-ctl start
fail: gitlab-workhorse: runsv not running
fail: logrotate: runsv not running
fail: nginx: runsv not running
fail: postgresql: runsv not running
fail: redis: runsv not running
fail: sidekiq: runsv not running
fail: unicorn: runsv not running
ps ax | grep git
(empty)
systemctl restart gitlab-runsvdir.service
systemctl status gitlab-runsvdir.service -l
● gitlab-runsvdir.service - GitLab Runit supervision process
Loaded: loaded (/usr/lib/systemd/system/gitlab-runsvdir.service; enabled; vendor preset: e
Drop-In: /etc/systemd/system/gitlab-runsvdir.service.d
└─requires.conf
Active: active (running) since Sat 2016-08-13 09:19:17 PDT; 3s ago
Main PID: 3543 (runsvdir)
Tasks: 46
Memory: 151.5M
CPU: 2.428s
CGroup: /system.slice/gitlab-runsvdir.service
├─3543 runsvdir -P /opt/gitlab/service log: ......................................
├─3545 runsv postgresql
Related issues
Edited by 🤖 GitLab Bot 🤖