unicorn workers slowly take up to 100% cpu

Summary

Over the course of approx two days the CPU usage of the unicorn workers slowly grows up to 100% at which point GitLab becomes unresponsive. Using gitlab-ee 9.4.5

Steps to reproduce

I didn't encounter this on gitlab 9.3.x. Nothing special needed to reproduce this here with gitlab 9.4. Just let it run for a while.

This also happened over the weekend so where we didn't really have any "traffic" on GitLab. Suspect this might have something to do with prometheus.

Relevant logs and/or screenshots

running the backtrace on a unicorn worker shows in /var/log/gitlab/unicorn/unicorn_stderr.log:

        from /opt/gitlab/embedded/bin/unicorn:22:in `<main>'
        from /opt/gitlab/embedded/bin/unicorn:22:in `load'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:658:in `worker_loop'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:562:in `process_client'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/urlmap.rb:50:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/urlmap.rb:50:in `each'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/urlmap.rb:66:in `block in call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/railtie.rb:194:in `method_missing'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/railtie.rb:194:in `public_send'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/application.rb:165:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/engine.rb:518:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sentry-raven-2.5.3/lib/raven/integrations/rack.rb:50:in `call'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/sendfile.rb:113:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/lock.rb:17:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/runtime.rb:18:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/methodoverride.rb:22:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/request_id.rb:21:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_context.rb:18:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/rack/logger.rb:20:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/tagged_logging.rb:68:in `tagged'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/tagged_logging.rb:26:in `tagged'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/tagged_logging.rb:68:in `block in tagged'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/rack/logger.rb:20:in `block in call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/rack/logger.rb:38:in `call_app'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/query_cache.rb:36:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/cookies.rb:560:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:220:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/session/abstract/id.rb:225:in `context'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/flash.rb:260:in `call'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/readonly_geo.rb:30:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/head.rb:13:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/conditionalget.rb:25:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-1.6.5/lib/rack/etag.rb:24:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/rack-cors-0.4.0/lib/rack/cors.rb:80:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/warden-1.2.6/lib/warden/manager.rb:34:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/warden-1.2.6/lib/warden/manager.rb:34:in `catch'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/warden-1.2.6/lib/warden/manager.rb:35:in `block in call'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:29:in `call'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/transaction.rb:49:in `run'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:29:in `block in call'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rails_queue_duration.rb:20:in `call'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:11:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/flipper-0.10.2/lib/flipper/middleware/memoizer.rb:55:in `call'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:16:in `call'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:14:in `call'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:93:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omniauth-1.4.2/lib/omniauth/strategy.rb:164:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omniauth-1.4.2/lib/omniauth/strategy.rb:186:in `call!'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/performance_bar/peek_performance_bar_with_rack_body.rb:16:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/routing/route_set.rb:817:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/journey/router.rb:30:in `serve'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/journey/router.rb:30:in `each'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/journey/router.rb:43:in `block in serve'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/routing/route_set.rb:43:in `serve'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_controller/metal.rb:237:in `block in action'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_controller/metal.rb:196:in `dispatch'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionview-4.2.8/lib/action_view/rendering.rb:30:in `process'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/abstract_controller/base.rb:137:in `process'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications.rb:164:in `instrument'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications.rb:164:in `block in instrument'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/abstract_controller/callbacks.rb:19:in `process_action'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:505:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:497:in `block in around'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:298:in `block in halting_and_conditional'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:505:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:117:in `call'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/abstract_controller/base.rb:198:in `process_action'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
        from /opt/gitlab/embedded/service/gitlab-rails/app/controllers/metrics_controller.rb:9:in `index'
        from /opt/gitlab/embedded/service/gitlab-rails/app/services/metrics_service.rb:21:in `metrics_text'
        from /opt/gitlab/embedded/service/gitlab-rails/app/services/metrics_service.rb:11:in `prometheus_metrics_text'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta11/lib/prometheus/client/formats/text.rb:40:in `marshal_multiprocess'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta11/lib/prometheus/client/formats/text.rb:97:in `load_metrics'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta11/lib/prometheus/client/formats/text.rb:97:in `each'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta11/lib/prometheus/client/formats/text.rb:103:in `block in load_metrics'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta11/lib/prometheus/client/formats/text.rb:103:in `each'
        from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta11/lib/prometheus/client/formats/text.rb:117:in `block (2 levels) in load_metrics'

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.6.6
Bundler Version:1.13.7
Rake Version:	10.5.0
Redis Version:	3.2.5
Git Version:	2.13.4
Sidekiq Version:5.0.0
Go Version:	unknown

GitLab information
Version:	9.4.5-ee
Revision:	00ce12f
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
DB Version:	9.6.3
URL:		https://gitlab.roboception.de
HTTP Clone URL:	https://gitlab.roboception.de/some-group/some-project.git
SSH Clone URL:	git@gitlab.roboception.de:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	yes
Using Omniauth:	no


GitLab Shell
Version:	5.3.1
Repository storage paths:
  • default: /mnt/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check while the workers are stuck at 100% cpu 
Checking GitLab Shell ...

GitLab Shell version >= 5.3.1 ? ... OK (5.3.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
5/2 ... ok
8/3 ... ok
8/4 ... ok
8/5 ... ok
9/6 ... ok
10/7 ... ok
8/8 ... ok
10/9 ... ok
10/10 ... ok
4/11 ... ok
9/12 ... ok
10/13 ... ok
10/14 ... ok
12/15 ... ok
12/16 ... ok
12/17 ... ok
12/18 ... ok
12/19 ... ok
12/20 ... ok
12/21 ... ok
10/22 ... ok
8/23 ... ok
9/24 ... ok
10/25 ... ok
5/26 ... ok
12/27 ... ok
12/28 ... ok
10/29 ... ok
10/30 ... ok
12/31 ... ok
12/32 ... ok
4/33 ... ok
10/34 ... ok
13/35 ... ok
8/36 ... ok
5/37 ... ok
10/38 ... ok
9/39 ... ok
8/40 ... ok
12/41 ... ok
4/42 ... ok
9/43 ... ok
12/44 ... ok
5/45 ... ok
5/46 ... ok
10/47 ... ok
10/48 ... ok
12/49 ... ok
12/50 ... ok
16/51 ... ok
16/52 ... ok
16/53 ... ok
16/54 ... ok
12/55 ... ok
21/56 ... ok
23/57 ... ok
15/58 ... ok
7/59 ... ok
10/60 ... ok
10/61 ... ok
18/62 ... ok
10/63 ... ok
9/64 ... ok
12/65 ... ok
10/66 ... ok
26/67 ... ok
26/68 ... ok
26/69 ... ok
12/70 ... ok
12/71 ... ok
10/72 ... ok
12/73 ... ok
10/75 ... ok
23/76 ... ok
12/77 ... ok
12/78 ... ok
10/79 ... ok
10/80 ... ok
12/81 ... ok
5/83 ... ok
10/84 ... ok
5/85 ... ok
21/86 ... ok
10/88 ... ok
12/89 ... ok
12/90 ... ok
12/91 ... repository is empty
12/92 ... ok
5/93 ... ok
12/94 ... ok
9/95 ... ok
23/96 ... ok
12/97 ... ok
5/98 ... ok
25/99 ... ok
12/100 ... ok
10/101 ... ok
12/102 ... ok
12/103 ... ok
7/105 ... ok
5/106 ... ok
10/107 ... ok
10/108 ... ok
10/109 ... ok
23/110 ... ok
10/111 ... ok
10/112 ... ok
5/114 ... ok
5/115 ... ok
10/116 ... ok
5/118 ... ok
21/119 ... ok
12/120 ... ok
12/122 ... ok
10/124 ... ok
10/125 ... ok
23/126 ... ok
12/127 ... ok
10/128 ... ok
5/130 ... ok
5/131 ... ok
10/132 ... ok
10/134 ... ok
5/135 ... ok
7/136 ... ok
5/137 ... ok
5/138 ... ok
12/139 ... ok
22/140 ... ok
22/141 ... ok
12/142 ... ok
12/143 ... ok
12/144 ... repository is empty
11/145 ... ok
19/146 ... ok
12/147 ... ok
27/148 ... ok
5/149 ... ok
12/150 ... ok
12/152 ... ok
10/153 ... ok
5/154 ... ok
12/155 ... ok
10/156 ... ok
10/157 ... ok
12/158 ... ok
10/159 ... ok
5/160 ... ok
10/161 ... ok
10/162 ... ok
12/163 ... ok
27/164 ... ok
5/165 ... ok
10/166 ... ok
22/167 ... ok
22/168 ... ok
10/169 ... ok
5/170 ... ok
10/171 ... ok
10/172 ... ok
11/173 ... ok
22/174 ... ok
27/175 ... ok
10/176 ... ok
12/177 ... ok
10/178 ... ok
10/179 ... ok
10/180 ... ok
10/181 ... ok
5/182 ... ok
12/183 ... ok
5/184 ... ok
5/185 ... ok
12/186 ... ok
10/187 ... ok
10/188 ... ok
12/189 ... ok
5/190 ... ok
10/191 ... ok
12/192 ... ok
19/193 ... ok
5/194 ... ok
23/195 ... ok
12/196 ... ok
11/197 ... ok
21/198 ... ok
12/199 ... ok
5/201 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: FAILED: Failed to connect to internal API
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
Please fix the error above and rerun the checks.


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Reply by email ...


Reply by email is disabled in config/gitlab.yml


Checking Reply by email ... Finished


Checking LDAP ...


Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
xxxxx


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
5/2 ... yes
8/3 ... yes
8/4 ... yes
8/5 ... yes
9/6 ... yes
10/7 ... yes
8/8 ... yes
10/9 ... yes
10/10 ... yes
4/11 ... yes
9/12 ... yes
10/13 ... yes
10/14 ... yes
12/15 ... yes
12/16 ... yes
12/17 ... yes
12/18 ... yes
12/19 ... yes
12/20 ... yes
12/21 ... yes
10/22 ... yes
8/23 ... yes
9/24 ... yes
10/25 ... yes
5/26 ... yes
12/27 ... yes
12/28 ... yes
10/29 ... yes
10/30 ... yes
12/31 ... yes
12/32 ... yes
4/33 ... yes
10/34 ... yes
13/35 ... yes
8/36 ... yes
5/37 ... yes
10/38 ... yes
9/39 ... yes
8/40 ... yes
12/41 ... yes
4/42 ... yes
9/43 ... yes
12/44 ... yes
5/45 ... yes
5/46 ... yes
10/47 ... yes
10/48 ... yes
12/49 ... yes
12/50 ... yes
16/51 ... yes
16/52 ... yes
16/53 ... yes
16/54 ... yes
12/55 ... yes
21/56 ... yes
23/57 ... yes
15/58 ... yes
7/59 ... yes
10/60 ... yes
10/61 ... yes
18/62 ... yes
10/63 ... yes
9/64 ... yes
12/65 ... yes
10/66 ... yes
26/67 ... yes
26/68 ... yes
26/69 ... yes
12/70 ... yes
12/71 ... yes
10/72 ... yes
12/73 ... yes
10/75 ... yes
23/76 ... yes
12/77 ... yes
12/78 ... yes
10/79 ... yes
10/80 ... yes
12/81 ... yes
5/83 ... yes
10/84 ... yes
5/85 ... yes
21/86 ... yes
10/88 ... yes
12/89 ... yes
12/90 ... yes
12/91 ... yes
12/92 ... yes
5/93 ... yes
12/94 ... yes
9/95 ... yes
23/96 ... yes
12/97 ... yes
5/98 ... yes
25/99 ... yes
12/100 ... yes
10/101 ... yes
12/102 ... yes
12/103 ... yes
7/105 ... yes
5/106 ... yes
10/107 ... yes
10/108 ... yes
10/109 ... yes
23/110 ... yes
10/111 ... yes
10/112 ... yes
5/114 ... yes
5/115 ... yes
10/116 ... yes
5/118 ... yes
21/119 ... yes
12/120 ... yes
12/122 ... yes
10/124 ... yes
10/125 ... yes
23/126 ... yes
12/127 ... yes
10/128 ... yes
5/130 ... yes
5/131 ... yes
10/132 ... yes
10/134 ... yes
5/135 ... yes
7/136 ... yes
5/137 ... yes
5/138 ... yes
12/139 ... yes
22/140 ... yes
22/141 ... yes
12/142 ... yes
12/143 ... yes
12/144 ... yes
11/145 ... yes
19/146 ... yes
12/147 ... yes
27/148 ... yes
5/149 ... yes
12/150 ... yes
12/152 ... yes
10/153 ... yes
5/154 ... yes
12/155 ... yes
10/156 ... yes
10/157 ... yes
12/158 ... yes
10/159 ... yes
5/160 ... yes
10/161 ... yes
10/162 ... yes
12/163 ... yes
27/164 ... yes
5/165 ... yes
10/166 ... yes
22/167 ... yes
22/168 ... yes
10/169 ... yes
5/170 ... yes
10/171 ... yes
10/172 ... yes
11/173 ... yes
22/174 ... yes
27/175 ... yes
10/176 ... yes
12/177 ... yes
10/178 ... yes
10/179 ... yes
10/180 ... yes
10/181 ... yes
5/182 ... yes
12/183 ... yes
5/184 ... yes
5/185 ... yes
12/186 ... yes
10/187 ... yes
10/188 ... yes
12/189 ... yes
5/190 ... yes
10/191 ... yes
12/192 ... yes
19/193 ... yes
5/194 ... yes
23/195 ... yes
12/196 ... yes
11/197 ... yes
21/198 ... yes
12/199 ... yes
5/201 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.3 ? ... yes (2.3.3)
Git version >= 2.7.3 ? ... yes (2.13.4)
Active users: ... 19
Elasticsearch version 5.1 - 5.3? ... no (5.5.0)
For more information see:
doc/integration/elasticsearch.md


Checking GitLab ... Finished
Expand for output related to the GitLab application check after a gitlab-ctl restart 
Checking GitLab Shell ...

GitLab Shell version >= 5.3.1 ? ... OK (5.3.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
5/2 ... ok
8/3 ... ok
8/4 ... ok
8/5 ... ok
9/6 ... ok
10/7 ... ok
8/8 ... ok
10/9 ... ok
10/10 ... ok
4/11 ... ok
9/12 ... ok
10/13 ... ok
10/14 ... ok
12/15 ... ok
12/16 ... ok
12/17 ... ok
12/18 ... ok
12/19 ... ok
12/20 ... ok
12/21 ... ok
10/22 ... ok
8/23 ... ok
9/24 ... ok
10/25 ... ok
5/26 ... ok
12/27 ... ok
12/28 ... ok
10/29 ... ok
10/30 ... ok
12/31 ... ok
12/32 ... ok
4/33 ... ok
10/34 ... ok
13/35 ... ok
8/36 ... ok
5/37 ... ok
10/38 ... ok
9/39 ... ok
8/40 ... ok
12/41 ... ok
4/42 ... ok
9/43 ... ok
12/44 ... ok
5/45 ... ok
5/46 ... ok
10/47 ... ok
10/48 ... ok
12/49 ... ok
12/50 ... ok
16/51 ... ok
16/52 ... ok
16/53 ... ok
16/54 ... ok
12/55 ... ok
21/56 ... ok
23/57 ... ok
15/58 ... ok
7/59 ... ok
10/60 ... ok
10/61 ... ok
18/62 ... ok
10/63 ... ok
9/64 ... ok
12/65 ... ok
10/66 ... ok
26/67 ... ok
26/68 ... ok
26/69 ... ok
12/70 ... ok
12/71 ... ok
10/72 ... ok
12/73 ... ok
10/75 ... ok
23/76 ... ok
12/77 ... ok
12/78 ... ok
10/79 ... ok
10/80 ... ok
12/81 ... ok
5/83 ... ok
10/84 ... ok
5/85 ... ok
21/86 ... ok
10/88 ... ok
12/89 ... ok
12/90 ... ok
12/91 ... repository is empty
12/92 ... ok
5/93 ... ok
12/94 ... ok
9/95 ... ok
23/96 ... ok
12/97 ... ok
5/98 ... ok
25/99 ... ok
12/100 ... ok
10/101 ... ok
12/102 ... ok
12/103 ... ok
7/105 ... ok
5/106 ... ok
10/107 ... ok
10/108 ... ok
10/109 ... ok
23/110 ... ok
10/111 ... ok
10/112 ... ok
5/114 ... ok
5/115 ... ok
10/116 ... ok
5/118 ... ok
21/119 ... ok
12/120 ... ok
12/122 ... ok
10/124 ... ok
10/125 ... ok
23/126 ... ok
12/127 ... ok
10/128 ... ok
5/130 ... ok
5/131 ... ok
10/132 ... ok
10/134 ... ok
5/135 ... ok
7/136 ... ok
5/137 ... ok
5/138 ... ok
12/139 ... ok
22/140 ... ok
22/141 ... ok
12/142 ... ok
12/143 ... ok
12/144 ... repository is empty
11/145 ... ok
19/146 ... ok
12/147 ... ok
27/148 ... ok
5/149 ... ok
12/150 ... ok
12/152 ... ok
10/153 ... ok
5/154 ... ok
12/155 ... ok
10/156 ... ok
10/157 ... ok
12/158 ... ok
10/159 ... ok
5/160 ... ok
10/161 ... ok
10/162 ... ok
12/163 ... ok
27/164 ... ok
5/165 ... ok
10/166 ... ok
22/167 ... ok
22/168 ... ok
10/169 ... ok
5/170 ... ok
10/171 ... ok
10/172 ... ok
11/173 ... ok
22/174 ... ok
27/175 ... ok
10/176 ... ok
12/177 ... ok
10/178 ... ok
10/179 ... ok
10/180 ... ok
10/181 ... ok
5/182 ... ok
12/183 ... ok
5/184 ... ok
5/185 ... ok
12/186 ... ok
10/187 ... ok
10/188 ... ok
12/189 ... ok
5/190 ... ok
10/191 ... ok
12/192 ... ok
19/193 ... ok
5/194 ... ok
23/195 ... ok
12/196 ... ok
11/197 ... ok
21/198 ... ok
12/199 ... ok
5/201 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Reply by email ...


Reply by email is disabled in config/gitlab.yml


Checking Reply by email ... Finished


Checking LDAP ...


Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
xxxxxx


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
5/2 ... yes
8/3 ... yes
8/4 ... yes
8/5 ... yes
9/6 ... yes
10/7 ... yes
8/8 ... yes
10/9 ... yes
10/10 ... yes
4/11 ... yes
9/12 ... yes
10/13 ... yes
10/14 ... yes
12/15 ... yes
12/16 ... yes
12/17 ... yes
12/18 ... yes
12/19 ... yes
12/20 ... yes
12/21 ... yes
10/22 ... yes
8/23 ... yes
9/24 ... yes
10/25 ... yes
5/26 ... yes
12/27 ... yes
12/28 ... yes
10/29 ... yes
10/30 ... yes
12/31 ... yes
12/32 ... yes
4/33 ... yes
10/34 ... yes
13/35 ... yes
8/36 ... yes
5/37 ... yes
10/38 ... yes
9/39 ... yes
8/40 ... yes
12/41 ... yes
4/42 ... yes
9/43 ... yes
12/44 ... yes
5/45 ... yes
5/46 ... yes
10/47 ... yes
10/48 ... yes
12/49 ... yes
12/50 ... yes
16/51 ... yes
16/52 ... yes
16/53 ... yes
16/54 ... yes
12/55 ... yes
21/56 ... yes
23/57 ... yes
15/58 ... yes
7/59 ... yes
10/60 ... yes
10/61 ... yes
18/62 ... yes
10/63 ... yes
9/64 ... yes
12/65 ... yes
10/66 ... yes
26/67 ... yes
26/68 ... yes
26/69 ... yes
12/70 ... yes
12/71 ... yes
10/72 ... yes
12/73 ... yes
10/75 ... yes
23/76 ... yes
12/77 ... yes
12/78 ... yes
10/79 ... yes
10/80 ... yes
12/81 ... yes
5/83 ... yes
10/84 ... yes
5/85 ... yes
21/86 ... yes
10/88 ... yes
12/89 ... yes
12/90 ... yes
12/91 ... yes
12/92 ... yes
5/93 ... yes
12/94 ... yes
9/95 ... yes
23/96 ... yes
12/97 ... yes
5/98 ... yes
25/99 ... yes
12/100 ... yes
10/101 ... yes
12/102 ... yes
12/103 ... yes
7/105 ... yes
5/106 ... yes
10/107 ... yes
10/108 ... yes
10/109 ... yes
23/110 ... yes
10/111 ... yes
10/112 ... yes
5/114 ... yes
5/115 ... yes
10/116 ... yes
5/118 ... yes
21/119 ... yes
12/120 ... yes
12/122 ... yes
10/124 ... yes
10/125 ... yes
23/126 ... yes
12/127 ... yes
10/128 ... yes
5/130 ... yes
5/131 ... yes
10/132 ... yes
10/134 ... yes
5/135 ... yes
7/136 ... yes
5/137 ... yes
5/138 ... yes
12/139 ... yes
22/140 ... yes
22/141 ... yes
12/142 ... yes
12/143 ... yes
12/144 ... yes
11/145 ... yes
19/146 ... yes
12/147 ... yes
27/148 ... yes
5/149 ... yes
12/150 ... yes
12/152 ... yes
10/153 ... yes
5/154 ... yes
12/155 ... yes
10/156 ... yes
10/157 ... yes
12/158 ... yes
10/159 ... yes
5/160 ... yes
10/161 ... yes
10/162 ... yes
12/163 ... yes
27/164 ... yes
5/165 ... yes
10/166 ... yes
22/167 ... yes
22/168 ... yes
10/169 ... yes
5/170 ... yes
10/171 ... yes
10/172 ... yes
11/173 ... yes
22/174 ... yes
27/175 ... yes
10/176 ... yes
12/177 ... yes
10/178 ... yes
10/179 ... yes
10/180 ... yes
10/181 ... yes
5/182 ... yes
12/183 ... yes
5/184 ... yes
5/185 ... yes
12/186 ... yes
10/187 ... yes
10/188 ... yes
12/189 ... yes
5/190 ... yes
10/191 ... yes
12/192 ... yes
19/193 ... yes
5/194 ... yes
23/195 ... yes
12/196 ... yes
11/197 ... yes
21/198 ... yes
12/199 ... yes
5/201 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.3 ? ... yes (2.3.3)
Git version >= 2.7.3 ? ... yes (2.13.4)
Active users: ... 19
Elasticsearch version 5.1 - 5.3? ... no (5.5.0)
For more information see:
doc/integration/elasticsearch.md


Checking GitLab ... Finished