Slow startup
Since upgrading to Gitlab-EE v16.6.0 startup is really slow. A restart used to take around 5 minutes, but now it is stuck for 3 hours with the following log:
Thank you for using GitLab Docker Image!
Current version: gitlab-ee=16.6.0-ee.0
Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:
docker exec -it gitlab editor /etc/gitlab/gitlab.rb
docker restart gitlab
For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
If this container fails to start due to permission problems try to fix it by executing:
docker exec -it gitlab update-permissions
docker restart gitlab
Cleaning stale PIDs & sockets
A ps
reveals that it is in /assets/wrapper
doing a find
over the whole data volume looking for stale PIDs and sockets.
function clean_stale_pids() {
# cleanup known pid/socket files
for x in /opt/gitlab/sv /run /var/opt/gitlab ; do
# find
# - any (s)ocket or regular (f)ile
# - by the name of "*.pid" or "socket.?"
# - and delete them
find $x \
\( \
-type f \
-o -type s \
\) \(\
-name pid \
-o -name "*.pid" \
-o -name "socket.?" \
\) \
-delete ;
done
}
Monitoring the find progress by looking at /proc/<pid_of_find/fd
further shows that is it spending the bulk of the time in /var/opt/gitlab/gitlab-rails/
in the (build) artifacts and container images directory. Manually running the find
on a running instance shows that all the found PIDs and sockets are not in the gitlab-rails
directory:
/var/opt/gitlab/gitaly/gitaly.pid
/var/opt/gitlab/postgresql/data/postmaster.pid
/opt/gitlab/sv/sshd/log/supervise/pid
/opt/gitlab/sv/sshd/supervise/pid
/opt/gitlab/sv/logrotate/log/supervise/pid
/opt/gitlab/sv/logrotate/supervise/pid
/opt/gitlab/sv/redis/log/supervise/pid
/opt/gitlab/sv/redis/supervise/pid
/opt/gitlab/sv/gitaly/log/supervise/pid
/opt/gitlab/sv/gitaly/supervise/pid
/opt/gitlab/sv/postgresql/log/supervise/pid
/opt/gitlab/sv/postgresql/supervise/pid
/opt/gitlab/sv/gitlab-kas/log/supervise/pid
/opt/gitlab/sv/gitlab-kas/supervise/pid
/opt/gitlab/sv/puma/log/supervise/pid
/opt/gitlab/sv/puma/supervise/pid
/opt/gitlab/sv/sidekiq/log/supervise/pid
/opt/gitlab/sv/sidekiq/supervise/pid
/opt/gitlab/sv/gitlab-workhorse/log/supervise/pid
/opt/gitlab/sv/gitlab-workhorse/supervise/pid
/opt/gitlab/sv/registry/log/supervise/pid
/opt/gitlab/sv/registry/supervise/pid
Maybe excluding gitlab-rails
could speed startup up significantly?
We are running the docker container version.
Edited by Sammy Atmadja