Gitlab fails to start on reboot cause of logging
centos 6 gitlab version: 9.4.7 and 10.5.5 Omnibus
While doing reboots, Gitlab does not start properly because it has issues seeing logs, the main issues I have seen is workhorse failing cause it can't send logs to svlogd. No logs are really showing this problem, the giltab check just says the API does not work and to try to correct the config file. Closest thing I can find to logs is the following:
root 4825 100 0.0 6136 484 ? R 16:13 0:21 runsvdir -P /opt/gitlab/service log: /nginx?svlogd: warning: processor failed, restart: /var/log/gitlab/gitlab-workhorse?svllogd: fatal: unable to open input for processor: /var/log/gitlab/mailroom: file does not exist?svlogd: warning: processor failed, restart: /var/log/gitlab/mailroom?svlogd: fatal: unable to open input for processor: /var/log/gitlab/nginx: file does not exist?svlogd: fatal: unable to open input for processor:
An strace of the process "svlogd -tt /var/log/gitlab/gitlab-workhorse" Shows the following
strace
fchdir(4) = 0
write(2, "svlogd: warning: processor faile"..., 77) = -1 EAGAIN (Resource temporarily unavailable)
unlink("@400000005ab143893644f79c.t") = -1 ENOENT (No such file or directory)
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f90a97ea9d0) = 22931
fchdir(3) = 0
rt_sigprocmask(SIG_UNBLOCK, [HUP], NULL, 8) = 0
wait4(22931, [{WIFEXITED(s) && WEXITSTATUS(s) == 111}], 0, NULL) = 22931
rt_sigprocmask(SIG_BLOCK, [HUP], NULL, 8) = 0
fchdir(4) = 0
write(2, "svlogd: warning: processor faile"..., 77) = -1 EAGAIN (Resource temporarily unavailable)
unlink("@400000005ab143893644f79c.t") = -1 ENOENT (No such file or directory)
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f90a97ea9d0) = 22958
fchdir(3) = 0Usually the following process will fix the issue
sudo gitlab-ctl stop
sudo initctl stop gitlab-runsvdir
sudo initctl start gitlab-runsvdir
sudo gitlab-ctl start
If this issue can not be resolved it would at least be nice to get an actual error in the gitlab check command to point people in the right direction.