`gitlab-rake gitlab:elastic:index` fails with ConnectionTimeoutError
Summary
Trying build an ElasticSearch index fails.
Steps to reproduce
Run gitlab-rake gitlab:elastic:index against a moderate-size installation (Haskell migration used here, gitlab-ce#55039).
Example Project
https://gitlab.staging.haskell.org/
What is the current bug behavior?
gitlab-rake gitlab:elastic:index fails with ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use.
What is the expected correct behavior?
Indexing should complete successfully.
Relevant logs and/or screenshots
$ gitlab-rake gitlab:elastic:index
/var/gitlab/state/config/application.rb:182: warning: already initialized constant Gitlab::Application::LOOSE_EE_APP_ASSETS
/nix/store/6rxqdcbkc8bi7i27darzy3pgcypwvz8p-gitlab-ee-11.7.0/share/gitlab/config/application.rb:182: warning: previous definition of LOOSE_EE_APP_ASSETS was here
WARNING: This version of GitLab depends on gitlab-shell 8.4.4, but you're running 8.4.3. Please update gitlab-shell.
W, [2019-02-06T18:16:48.548706 #3552] WARN -- : Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects.
W, [2019-02-06T18:16:48.548832 #3552] WARN -- : Creating scope :join_project. Overwriting existing method MergeRequest.join_project.
W, [2019-02-06T18:16:48.548875 #3552] WARN -- : Creating scope :references_project. Overwriting existing method MergeRequest.references_project.
W, [2019-02-06T18:16:48.561693 #3552] WARN -- : Creating scope :system. Overwriting existing method Note.system.
W, [2019-02-06T18:16:49.660707 #3552] WARN -- : Creating scope :opened. Overwriting existing method Epic.opened.
W, [2019-02-06T18:16:49.660796 #3552] WARN -- : Creating scope :closed. Overwriting existing method Epic.closed.
W, [2019-02-06T18:16:50.952542 #3552] WARN -- : Creating scope :push_hooks. Overwriting existing method GroupHook.push_hooks.
W, [2019-02-06T18:16:50.952608 #3552] WARN -- : Creating scope :tag_push_hooks. Overwriting existing method GroupHook.tag_push_hooks.
W, [2019-02-06T18:16:50.952660 #3552] WARN -- : Creating scope :issue_hooks. Overwriting existing method GroupHook.issue_hooks.
W, [2019-02-06T18:16:50.952708 #3552] WARN -- : Creating scope :confidential_issue_hooks. Overwriting existing method GroupHook.confidential_issue_hooks.
W, [2019-02-06T18:16:50.952754 #3552] WARN -- : Creating scope :note_hooks. Overwriting existing method GroupHook.note_hooks.
W, [2019-02-06T18:16:50.952798 #3552] WARN -- : Creating scope :merge_request_hooks. Overwriting existing method GroupHook.merge_request_hooks.
W, [2019-02-06T18:16:50.952843 #3552] WARN -- : Creating scope :job_hooks. Overwriting existing method GroupHook.job_hooks.
W, [2019-02-06T18:16:50.952869 #3552] WARN -- : Creating scope :pipeline_hooks. Overwriting existing method GroupHook.pipeline_hooks.
W, [2019-02-06T18:16:50.952905 #3552] WARN -- : Creating scope :wiki_page_hooks. Overwriting existing method GroupHook.wiki_page_hooks.
W, [2019-02-06T18:16:51.802669 #3552] WARN -- : Creating scope :created. Overwriting existing method Ci::Stage.created.
W, [2019-02-06T18:16:51.802776 #3552] WARN -- : Creating scope :pending. Overwriting existing method Ci::Stage.pending.
W, [2019-02-06T18:16:51.803351 #3552] WARN -- : Creating scope :running. Overwriting existing method Ci::Stage.running.
W, [2019-02-06T18:16:51.803432 #3552] WARN -- : Creating scope :success. Overwriting existing method Ci::Stage.success.
W, [2019-02-06T18:16:51.803532 #3552] WARN -- : Creating scope :failed. Overwriting existing method Ci::Stage.failed.
W, [2019-02-06T18:16:51.803614 #3552] WARN -- : Creating scope :canceled. Overwriting existing method Ci::Stage.canceled.
W, [2019-02-06T18:16:51.803679 #3552] WARN -- : Creating scope :skipped. Overwriting existing method Ci::Stage.skipped.
W, [2019-02-06T18:16:51.803724 #3552] WARN -- : Creating scope :manual. Overwriting existing method Ci::Stage.manual.
W, [2019-02-06T18:16:51.803790 #3552] WARN -- : Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled.
W, [2019-02-06T18:16:51.969079 #3552] WARN -- : Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc.
W, [2019-02-06T18:16:51.969147 #3552] WARN -- : Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc.
W, [2019-02-06T18:16:51.991937 #3552] WARN -- : Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.
W, [2019-02-06T18:16:51.992002 #3552] WARN -- : Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.
W, [2019-02-06T18:16:52.033076 #3552] WARN -- : Creating scope :fresh. Overwriting existing method ProjectSnippet.fresh.
W, [2019-02-06T18:16:52.035999 #3552] WARN -- : Creating scope :system. Overwriting existing method PrometheusMetric.system.
Index created
Index status has been reset
Indexing wiki of Glasgow Haskell Compiler / GHC...
Done!
Indexing wiki of David Eichmann / Glasgow Haskell Compiler...
Done!
I, [2019-02-06T18:17:11.973682 #3552] INFO -- : Indexing Milestones...
I, [2019-02-06T18:17:11.974049 #3552] INFO -- : Indexing Projects...
I, [2019-02-06T18:17:11.974167 #3552] INFO -- : Indexing Issues...
I, [2019-02-06T18:17:11.974341 #3552] INFO -- : Indexing MergeRequests...
I, [2019-02-06T18:17:11.974532 #3552] INFO -- : Indexing Snippets...
I, [2019-02-06T18:17:11.975599 #3552] INFO -- : Indexing Notes...
I, [2019-02-06T18:17:12.054308 #3552] INFO -- : Indexing Milestones... done
I, [2019-02-06T18:17:12.324669 #3552] INFO -- : Indexing MergeRequests... done
I, [2019-02-06T18:17:13.680420 #3552] INFO -- : Indexing Projects... done
I, [2019-02-06T18:17:14.010130 #3552] INFO -- : Indexing Snippets... done
I, [2019-02-06T18:20:36.096713 #3552] INFO -- : Indexing Notes... done
rake aborted!
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:202:in `block in wait_poll'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:193:in `loop'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:193:in `wait_poll'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:154:in `internal_poll'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:278:in `internal_poll'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:148:in `block in poll'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:158:in `synchronize'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:148:in `poll'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:717:in `acquire_connection'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_handling.rb:128:in `retrieve_connection'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_handling.rb:91:in `connection'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/model_schema.rb:228:in `quoted_table_name'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/relation/delegation.rb:43:in `quoted_table_name'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/relation/batches.rb:230:in `batch_order'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/relation/batches.rb:194:in `in_batches'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/relation/batches.rb:120:in `find_in_batches'
/nix/store/8gsj3cw3pjlwk03w62gynpp453ihv5i5-ruby2.5.3-activerecord-5.0.7.1/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/querying.rb:9:in `find_in_batches'
/nix/store/7mi9fzzwvq8k2y68dwdxwh2xrgvad9wq-ruby2.5.3-elasticsearch-model-0.1.9/lib/ruby/gems/2.5.0/gems/elasticsearch-model-0.1.9/lib/elasticsearch/model/proxy.rb:90:in `method_missing'
/nix/store/7mi9fzzwvq8k2y68dwdxwh2xrgvad9wq-ruby2.5.3-elasticsearch-model-0.1.9/lib/ruby/gems/2.5.0/gems/elasticsearch-model-0.1.9/lib/elasticsearch/model/adapters/active_record.rb:102:in `__find_in_batches'
/nix/store/7mi9fzzwvq8k2y68dwdxwh2xrgvad9wq-ruby2.5.3-elasticsearch-model-0.1.9/lib/ruby/gems/2.5.0/gems/elasticsearch-model-0.1.9/lib/elasticsearch/model/importing.rb:122:in `import'
/nix/store/7mi9fzzwvq8k2y68dwdxwh2xrgvad9wq-ruby2.5.3-elasticsearch-model-0.1.9/lib/ruby/gems/2.5.0/gems/elasticsearch-model-0.1.9/lib/elasticsearch/model.rb:116:in `import'
/nix/store/6rxqdcbkc8bi7i27darzy3pgcypwvz8p-gitlab-ee-11.7.0/share/gitlab/ee/app/models/concerns/elastic/application_search.rb:289:in `es_import'
/nix/store/6rxqdcbkc8bi7i27darzy3pgcypwvz8p-gitlab-ee-11.7.0/share/gitlab/ee/lib/tasks/gitlab/elastic.rake:80:in `block (4 levels) in <main>'
Tasks: TOP => gitlab:elastic:index_database => gitlab:elastic:index_issues
(See full trace by running task with --trace)Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: root Using RVM: no Ruby Version: unknown Gem Version: Bundler Version:1.17.2 Rake Version: 12.3.2 Redis Version: unknown Git Version: 2.19.2 Sidekiq Version:5.2.3 Go Version: unknownGitLab information Version: 11.7.0-ee Revision: 044fcce63e5 Directory: /nix/store/6rxqdcbkc8bi7i27darzy3pgcypwvz8p-gitlab-ee-11.7.0/share/gitlab DB Adapter: postgresql DB Version: 9.6.11 URL: https://gitlab.staging.haskell.org HTTP Clone URL: https://gitlab.staging.haskell.org/some-group/some-project.git SSH Clone URL: git@gitlab.staging.haskell.org:some-group/some-project.git Elasticsearch: yes Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: github
GitLab Shell Version: 8.4.3 Repository storage paths:
- default: /var/gitlab/state/repositories Hooks: /var/gitlab/state/shell/hooks Git: git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 8.4.4 ? ... FAIL. Please update gitlab-shell to 8.4.4 from 8.4.3 Running /nix/store/24l1w0q0lah4f4lnwmdfl1zxid66dij0-gitlab-shell-8.4.3/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/gitlab/state/home/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no Try fixing it: sudo -u git -H RAILS_ENV=production bin/background_jobs start For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... no Try fixing it: sudo chown -R git /var/gitlab/state/uploads sudo find /var/gitlab/state/uploads -type f -exec chmod 0644 {} ; sudo find /var/gitlab/state/uploads -type d -not -path /var/gitlab/state/uploads -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... no Try fixing it: Install the init script For more information see: doc/install/installation.md in section "Install Init Script" Please fix the error above and rerun the checks. Init script up-to-date? ... can't check because of previous errors Projects have namespace: ... 2/1 ... yes 2/4 ... yes 2/5 ... yes 2/6 ... yes 2/8 ... yes 2/9 ... yes 2/10 ... yes 4/11 ... yes 4/12 ... yes 4/13 ... yes 4/14 ... yes 4/15 ... yes 4/16 ... yes 4/17 ... yes 4/18 ... yes 4/19 ... yes 4/20 ... yes 4/21 ... yes 4/22 ... yes 4/23 ... yes 4/24 ... yes 4/25 ... yes 4/26 ... yes 4/27 ... yes 4/28 ... yes 4/29 ... yes 4/30 ... yes 4/31 ... yes 4/32 ... yes 4/33 ... yes 3/34 ... yes 11/35 ... yes 13/36 ... yes 14/37 ... yes 18/38 ... yes 23/39 ... yes 28/40 ... yes 7/41 ... yes 15/43 ... yes
29/44 ... yes
38/45 ... yes
40/46 ... yes
42/47 ... yes
36/48 ... yes
44/49 ... yes
45/50 ... yes
41/51 ... yes
46/52 ... yes
53/53 ... yes
65/54 ... yes
2/55 ... yes 68/56 ... yes 73/57 ... yes 76/58 ... yes 83/59 ... yes 52/60 ... yes 91/61 ... yes 101/63 ... yes 112/64 ... yes 12/65 ... yes 115/66 ... yes 35/68 ... yes 24/69 ... yes 141/70 ... yes 8/71 ... yes 54/72 ... yes 129/73 ... yes 145/74 ... yes 149/75 ... yes 149/76 ... yes 144/77 ... yes 2/78 ... yes 159/79 ... yes 161/80 ... yes 164/81 ... yes 135/82 ... yes 138/83 ... yes 48/84 ... yes 138/85 ... yes 51/86 ... yes 51/87 ... yes 10/88 ... yes 183/89 ... yes 184/90 ... yes 185/92 ... yes 199/93 ... yes 205/94 ... yes 4/95 ... yes 4/96 ... yes 4/97 ... yes 4/98 ... yes 4/99 ... yes 4/100 ... yes 4/101 ... yes 4/102 ... yes 4/103 ... yes 4/104 ... yes
2/105 ... yes 4/106 ... yes
4/107 ... yes 4/108 ... yes
4/109 ... yes 2/110 ... yes
4/111 ... yes 4/112 ... yes
212/113 ... yes 214/114 ... yes
15/116 ... yes 2/117 ... yes
219/118 ... yes 6/119 ... yes
6/120 ... yes 134/121 ... yes
3/122 ... yes
21/123 ... yes
241/124 ... yes
242/125 ... yes
221/126 ... yes
244/127 ... yes
14/128 ... yes
118/129 ... yes
50/130 ... yes
195/131 ... yes
265/133 ... yes 262/134 ... yes
3/135 ... yes
Redis version >= 2.8.0? ... no
Try fixing it:
Update your redis server to a version >= 2.8.0 For more information see:
gitlab-public-wiki/wiki/Trouble-Shooting-Guide in section sidekiq Please fix the error above and rerun the checks. Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.19.2) Git user has default SSH configuration? ... yes
Active users: ... 3416
Elasticsearch version 5.6 - 6.x? ... yes (6.5.1)Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished