bundle processes are killed/turned over too frequently

Summary

As a sys admin of an internal gitlab server, I see that the bundle processes are killed off frequently and re-created, and this causes unnecessary system slowness to the point of being almost unusable for pulling up label names or milestone names or any other RPC lookup.

Steps to reproduce

Use top or ps and watch the PID values turn over for any bundle proc under unicorn.

Example Project

Any internal gitlab server would do. It appears to have been from this:

https://about.gitlab.com/2015/06/05/how-gitlab-uses-unicorn-and-unicorn-worker-killer/

where the "answer" to memory leaks is to not fix them and instead just restart the process. Yikes!

What is the current bug behavior?

The bundle processes turn over frequently causes slowness in the UI due to the cost of the bundle process start time (context switch).

What is the expected correct behavior?

The bundle processes should NOT turn over frequently.

Relevant logs and/or screenshots

I increased the max memory here to 500 from 350:

# This file is used by Rack-based servers to start the application.

if defined?(Unicorn)
  require 'unicorn'

  if ENV['RAILS_ENV'] == 'production' || ENV['RAILS_ENV'] == 'staging'
    # Unicorn self-process killer
    require 'unicorn/worker_killer'

    min = (ENV['GITLAB_UNICORN_MEMORY_MIN'] || 300 * 1 << 20).to_i
    max = (ENV['GITLAB_UNICORN_MEMORY_MAX'] || 500 * 1 << 20).to_i

    # Max memory size (RSS) per worker
    use Unicorn::WorkerKiller::Oom, min, max
  end
end

require ::File.expand_path('../config/environment',  __FILE__)

map ENV['RAILS_RELATIVE_URL_ROOT'] || "/" do
  run Gitlab::Application
end

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info

[root@gitlab gitlab]# sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
System information
System:
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:
Bundler Version:1.15.4
Rake Version:	12.1.0
Redis Version:	3.0.3
Git Version:	2.9.3
Sidekiq Version:5.0.4
Go Version:	go1.8.3 linux/amd64
GitLab information
Version:	10.2.3-ee
Revision:	a0f9e22
Directory:	/home/git/gitlab
DB Adapter:	postgresql
DB Version:	9.4.9
URL:		http://gitlab.medstrat.com
HTTP Clone URL:	http://gitlab.medstrat.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.medstrat.com:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	no
GitLab Shell
Version:	5.9.4
Repository storage paths:

• default: /home/git/repositories Hooks: /home/git/gitlab-shell/hooks Git: /usr/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

TODO: but yes, tests I'm sure are passing.
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

Change config.ru from this

    max = (ENV['GITLAB_UNICORN_MEMORY_MAX'] || 350 * 1 << 20).to_i

to this:

    max = (ENV['GITLAB_UNICORN_MEMORY_MAX'] || 500 * 1 << 20).to_i

which is what I currently do after every gitlab update now (which is tedious to remember to do).