Error indexing elastic search on SElinux

Summary

Error indexing elastic search on SElinux

Steps to reproduce

  • Restore (or possibly install anew) the latest (11.11.0) version
  • install elasticsearch 6.8.0
  • verify curl localhost:9200
  • run sudo gitlab-rake gitlab:elastic:index (as per instructions)

What is the current bug behavior?

Error

** Invoke gitlab:elastic:index (first_time)
** Execute gitlab:elastic:index
** Invoke gitlab:elastic:create_empty_index (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:elastic:create_empty_index
rake aborted!
Errno::EHOSTUNREACH: No route to host - No route to host - connect(2) for "::1" port 9200 (localhost:9200)
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:623:in `rescue in create_socket'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:607:in `create_socket'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:755:in `block in connect'
/opt/gitlab/embedded/lib/ruby/2.5.0/timeout.rb:93:in `block in timeout'
/opt/gitlab/embedded/lib/ruby/2.5.0/timeout.rb:103:in `timeout'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:748:in `connect'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:511:in `query'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:177:in `query'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1242:in `do_get_block'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1019:in `block in do_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1133:in `protect_keep_alive_disconnected'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1014:in `do_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:856:in `request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/faraday-0.12.2/lib/faraday/adapter/httpclient.rb:38:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/faraday-0.12.2/lib/faraday/rack_builder.rb:141:in `build_response'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/faraday-0.12.2/lib/faraday/connection.rb:386:in `run_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/transport/http/faraday.rb:23:in `block in perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/transport/base.rb:261:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/client.rb:131:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/elasticsearch-api-5.0.3/lib/elasticsearch/api/actions/info.rb:15:in `info'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/helper.rb:29:in `create_empty_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:163:in `create_empty_index'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:104:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:10:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
Errno::EHOSTUNREACH: No route to host - connect(2) for "::1" port 9200
/opt/gitlab/embedded/lib/ruby/2.5.0/resolv-replace.rb:25:in `initialize'
/opt/gitlab/embedded/lib/ruby/2.5.0/resolv-replace.rb:25:in `initialize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:611:in `new'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:611:in `create_socket'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:755:in `block in connect'
/opt/gitlab/embedded/lib/ruby/2.5.0/timeout.rb:93:in `block in timeout'
/opt/gitlab/embedded/lib/ruby/2.5.0/timeout.rb:103:in `timeout'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:748:in `connect'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:511:in `query'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:177:in `query'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1242:in `do_get_block'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1019:in `block in do_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1133:in `protect_keep_alive_disconnected'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:1014:in `do_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httpclient-2.8.3/lib/httpclient.rb:856:in `request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/faraday-0.12.2/lib/faraday/adapter/httpclient.rb:38:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/faraday-0.12.2/lib/faraday/rack_builder.rb:141:in `build_response'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/faraday-0.12.2/lib/faraday/connection.rb:386:in `run_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/transport/http/faraday.rb:23:in `block in perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/transport/base.rb:261:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/elasticsearch-transport-5.0.3/lib/elasticsearch/transport/client.rb:131:in `perform_request'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/elasticsearch-api-5.0.3/lib/elasticsearch/api/actions/info.rb:15:in `info'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/helper.rb:29:in `create_empty_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:163:in `create_empty_index'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:104:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:10:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:elastic:create_empty_index

What is the expected correct behavior?

Successful indexation

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         CentOS 7.6.1810
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.21.0
Sidekiq Version:5.2.7
Go Version:     unknown


GitLab information
Version:        11.11.0-ee
Revision:       deb6f84e91f
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     9.6.11
URL:            https://gitlab.dbogatov.org
HTTP Clone URL: https://gitlab.dbogatov.org/some-group/some-project.git
SSH Clone URL:  git@gitlab.dbogatov.org:some-group/some-project.git
Elasticsearch:  yes
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: github, bitbucket, gitlab, google_oauth2, facebook, twitter


GitLab Shell
Version:        9.1.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Checking Reply by email ...


IMAP server credentials are correct? ... yes
Init.d configured correctly? ... skipped
MailRoom running? ... skipped


Checking Reply by email ... Finished


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
13/2 ... yes
2/3 ... yes
2/4 ... yes
2/5 ... yes
2/6 ... yes
57/7 ... yes
2/8 ... yes
2/9 ... yes
2/11 ... yes
2/13 ... yes
2/15 ... yes
57/16 ... yes
58/17 ... yes
58/22 ... yes
57/23 ... yes
2/24 ... yes
2/25 ... yes
2/26 ... yes
2/27 ... yes
14/28 ... yes
2/29 ... yes
2/30 ... yes
57/31 ... yes
2/32 ... yes
2/33 ... yes
14/34 ... yes
2/35 ... yes
2/36 ... yes
57/37 ... yes
2/38 ... yes
2/39 ... yes
57/40 ... yes
26/41 ... yes
26/42 ... yes
2/44 ... yes
2/47 ... yes
2/49 ... yes
29/50 ... yes
29/53 ... yes
2/54 ... yes
30/55 ... yes
30/56 ... yes
2/59 ... yes
19/60 ... yes
19/61 ... yes
39/62 ... yes
30/63 ... yes
39/65 ... yes
30/66 ... yes
2/67 ... yes
42/68 ... yes
42/69 ... yes
30/70 ... yes
45/71 ... yes
26/72 ... yes
30/73 ... yes
55/74 ... yes
45/75 ... yes
72/76 ... yes
2/77 ... yes
45/78 ... yes
2/79 ... yes
2/80 ... yes
26/81 ... yes
26/82 ... yes
54/83 ... yes
55/86 ... yes
54/87 ... yes
2/93 ... yes
54/94 ... yes
61/95 ... yes
2/96 ... yes
57/97 ... yes
57/98 ... yes
57/99 ... yes
57/100 ... yes
57/101 ... yes
57/102 ... yes
64/103 ... yes
24/104 ... yes
60/105 ... yes
60/106 ... yes
63/107 ... yes
61/108 ... yes
63/109 ... yes
39/111 ... yes
39/112 ... yes
2/113 ... yes
64/114 ... yes
63/115 ... yes
65/116 ... yes
65/119 ... yes
2/121 ... yes
65/122 ... yes
30/123 ... yes
63/125 ... yes
61/126 ... yes
61/127 ... yes
2/128 ... yes
54/129 ... yes
55/132 ... yes
30/133 ... yes
55/134 ... yes
67/135 ... yes
67/136 ... yes
69/138 ... yes
67/139 ... yes
68/140 ... yes
68/141 ... yes
39/142 ... yes
68/143 ... yes
69/144 ... yes
72/145 ... yes
2/146 ... yes
72/147 ... yes
57/98 ... yes
57/99 ... yes
57/100 ... yes
57/101 ... yes
57/102 ... yes
64/103 ... yes
24/104 ... yes
60/105 ... yes
60/106 ... yes
63/107 ... yes
61/108 ... yes
63/109 ... yes
39/111 ... yes
39/112 ... yes
2/113 ... yes
64/114 ... yes
63/115 ... yes
65/116 ... yes
65/119 ... yes
2/121 ... yes
65/122 ... yes
30/123 ... yes
63/125 ... yes
61/126 ... yes
61/127 ... yes
2/128 ... yes
54/129 ... yes
55/132 ... yes
30/133 ... yes
55/134 ... yes
67/135 ... yes
67/136 ... yes
69/138 ... yes
67/139 ... yes
68/140 ... yes
68/141 ... yes
39/142 ... yes
68/143 ... yes
69/144 ... yes
72/145 ... yes
2/146 ... yes
72/147 ... yes
67/148 ... yes
65/150 ... yes
65/156 ... yes
65/157 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.5.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 21
Elasticsearch version 5.6 - 6.x? ... Exception: No route to host - No route to host - connect(2) for "::1" port 9200 (localhost:9200)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Additional data

elastic search config

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: cs-test4.bu.edu
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: localhost
#network.bind_host: 0.0.0.0
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

sudo netstat -plntu

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5414            0.0.0.0:*               LISTEN      3652/rpc.statd
tcp        0      0 127.0.0.1:5000          0.0.0.0:*               LISTEN      14693/registry
tcp        0      0 127.0.0.1:9229          0.0.0.0:*               LISTEN      14696/gitlab-workho
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd
tcp        0      0 127.0.0.1:9200          0.0.0.0:*               LISTEN      21796/java
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      14932/unicorn maste
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      14697/nginx: master
tcp        0      0 127.0.0.1:8082          0.0.0.0:*               LISTEN      14680/sidekiq 5.2.7
tcp        0      0 127.0.0.1:9300          0.0.0.0:*               LISTEN      21796/java
tcp        0      0 127.0.0.1:9236          0.0.0.0:*               LISTEN      14770/gitaly
tcp        0      0 0.0.0.0:3606            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3638/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      4133/master
tcp        0      0 127.0.0.1:8090          0.0.0.0:*               LISTEN      14699/gitlab-pages
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      14697/nginx: master
tcp        0      0 0.0.0.0:8060            0.0.0.0:*               LISTEN      14697/nginx: master
tcp        0      0 127.0.0.1:8065          0.0.0.0:*               LISTEN      14688/mattermost
tcp6       0      0 :::5414                 :::*                    LISTEN      3652/rpc.statd
tcp6       0      0 :::111                  :::*                    LISTEN      3052/rpcbind
tcp6       0      0 :::3606                 :::*                    LISTEN      -
udp        0      0 0.0.0.0:5414            0.0.0.0:*                           3652/rpc.statd
udp        0      0 0.0.0.0:3606            0.0.0.0:*                           -
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/systemd
udp        0      0 127.0.0.1:323           0.0.0.0:*                           3076/chronyd
udp        0      0 0.0.0.0:682             0.0.0.0:*                           3052/rpcbind
udp        0      0 127.0.0.1:863           0.0.0.0:*                           3652/rpc.statd
udp6       0      0 :::5414                 :::*                                3652/rpc.statd
udp6       0      0 :::3606                 :::*                                -
udp6       0      0 :::111                  :::*                                3052/rpcbind
udp6       0      0 ::1:323                 :::*                                3076/chronyd
udp6       0      0 :::682                  :::*                                3052/rpcbind

Possible fixes

There is something with elasticsearch. I can do curl localhost:9200 but it fails when Gitlab tries to connect.

  • Can it happen due to SELinux policies?
  • Is it about how elastic search binds to host?