bundle processes are killed/turned over too frequently
Summary
As a sys admin of an internal gitlab server, I see that the bundle
processes are killed off frequently and re-created, and this causes unnecessary system slowness to the point of being almost unusable for pulling up label names or milestone names or any other RPC lookup.
Steps to reproduce
Use top
or ps
and watch the PID values turn over for any bundle
proc under unicorn
.
Example Project
Any internal gitlab server would do. It appears to have been from this:
https://about.gitlab.com/2015/06/05/how-gitlab-uses-unicorn-and-unicorn-worker-killer/
where the "answer" to memory leaks is to not fix them and instead just restart the process. Yikes!
What is the current bug behavior?
The bundle
processes turn over frequently causes slowness in the UI due to the cost of the bundle
process start time (context switch).
What is the expected correct behavior?
The bundle
processes should NOT turn over frequently.
Relevant logs and/or screenshots
I increased the max memory here to 500
from 350
:
# This file is used by Rack-based servers to start the application.
if defined?(Unicorn)
require 'unicorn'
if ENV['RAILS_ENV'] == 'production' || ENV['RAILS_ENV'] == 'staging'
# Unicorn self-process killer
require 'unicorn/worker_killer'
min = (ENV['GITLAB_UNICORN_MEMORY_MIN'] || 300 * 1 << 20).to_i
max = (ENV['GITLAB_UNICORN_MEMORY_MAX'] || 500 * 1 << 20).to_i
# Max memory size (RSS) per worker
use Unicorn::WorkerKiller::Oom, min, max
end
end
require ::File.expand_path('../config/environment', __FILE__)
map ENV['RAILS_RELATIVE_URL_ROOT'] || "/" do
run Gitlab::Application
end
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
[root@gitlab gitlab]# sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=productionSystem information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: Bundler Version:1.15.4 Rake Version: 12.1.0 Redis Version: 3.0.3 Git Version: 2.9.3 Sidekiq Version:5.0.4 Go Version: go1.8.3 linux/amd64
GitLab information Version: 10.2.3-ee Revision: a0f9e22 Directory: /home/git/gitlab DB Adapter: postgresql DB Version: 9.4.9 URL: http://gitlab.medstrat.com HTTP Clone URL: http://gitlab.medstrat.com/some-group/some-project.git SSH Clone URL: git@gitlab.medstrat.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: no
GitLab Shell Version: 5.9.4 Repository storage paths:
- default: /home/git/repositories Hooks: /home/git/gitlab-shell/hooks Git: /usr/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
TODO: but yes, tests I'm sure are passing.
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
Change config.ru
from this
max = (ENV['GITLAB_UNICORN_MEMORY_MAX'] || 350 * 1 << 20).to_i
to this:
max = (ENV['GITLAB_UNICORN_MEMORY_MAX'] || 500 * 1 << 20).to_i
which is what I currently do after every gitlab update now (which is tedious to remember to do).