External Redis Sentinel Cluster NOAUTH Authentication required
Summary
I've tried to migrate from Redis shipped with omnibus to a Redis Cluster with Sentinel when I've disabled the included Redis, Gitlab started to complain about authentication with the redis cluster. I've followed this documentation (I know that is for HA, but is the only one which includes Sentinel). I've also tried to add the redis-master ip because of this comment on Sentinel part.
####! To have Sentinel working, you must enable Redis TCP connection support
####! above and define a few Sentinel hosts below (to get a reliable setup
####! at least 3 hosts).
Steps to reproduce
Disable the bundled Redis with a cluster (see configuration part), reconfigure.
What is the current bug behavior?
Error 500 on web interface, gitlab sidekiq and workhorse complains, gitlab-rails errors.
What is the expected correct behavior?
Gitlab up and running
Relevant logs
Relevant logs
==> /var/log/gitlab/gitlab-rails/production.log <== Cached record for ApplicationSetting couldn't be loaded, falling back to uncached record: NOAUTH Authentication required. Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects. Creating scope :join_project. Overwriting existing method MergeRequest.join_project. Creating scope :references_project. Overwriting existing method MergeRequest.references_project. Creating scope :system. Overwriting existing method Note.system. Started GET "/-/metrics" for 127.0.0.1 at 2019-02-18 14:12:41 +0000 Cached record for ApplicationSetting couldn't be loaded, falling back to uncached record: NOAUTH Authentication required. Cached record for ApplicationSetting couldn't be loaded, falling back to uncached record: NOAUTH Authentication required. Cached record for ApplicationSetting couldn't be loaded, falling back to uncached record: NOAUTH Authentication required. Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.9ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)Redis::CommandError (NOAUTH Authentication required.):
lib/gitlab/middleware/multipart.rb:103:in
call' lib/gitlab/request_profiler/middleware.rb:16:in
call' ee/lib/gitlab/jira/middleware.rb:15:incall' lib/gitlab/middleware/go.rb:20:in
call' lib/gitlab/etag_caching/middleware.rb:13:incall' lib/gitlab/middleware/correlation_id.rb:16:in
block in call' lib/gitlab/correlation_id.rb:15:inuse_id' lib/gitlab/middleware/correlation_id.rb:15:in
call' lib/gitlab/middleware/read_only/controller.rb:42:incall' lib/gitlab/middleware/read_only.rb:18:in
call' lib/gitlab/middleware/basic_health_check.rb:25:incall' lib/gitlab/request_context.rb:20:in
call' lib/gitlab/metrics/requests_rack_middleware.rb:29:incall' lib/gitlab/middleware/release_env.rb:13:in
call'==> /var/log/gitlab/gitlab-rails/sidekiq.log <== 2019-02-18_14:12:42.79809 2019-02-18T14:12:42.797Z 30172 TID-ovv78115w INFO: Booting Sidekiq 5.2.3 with redis options {:scheme=>"redis", :host=>"redis-master", :port=>6379, :password=>"REDACTED", :sentinels=>[{:host=>"10.0.0.35", :port=
26379}, {:host=>"10.0.0.9", :port=>26379}, {:host=>"10.0.0.6", :port=>26379}], :namespace=>"resque:gitlab", :id=>"Sidekiq-server-PID-30172", :url=>nil} 2019-02-18_14:12:42.80975 NOAUTH Authentication required.
Details of package version
Provide the package version installation details
gitlab-ee-11.7.5-ee.0.el7.x86_64
Environment details
- Operating System:
CentOS 7
- Installation Target, remove incorrect values
- VM: Openstack
- Installation Type, remove incorrect values:
- New Installation
- Is there any other software running on the machine:
runners on different vms
- Is this a single or multiple node installation? Single
- Resources
- CPU:
4
- Memory total:
8gb
- CPU:
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
gitlab_rails['redis_host'] = "10.0.0.35" gitlab_rails['redis_port'] = 6379 gitlab_rails['redis_password'] = '[REDACTED]'gitlab_rails['redis_sentinels'] = [ {'host' => '10.0.0.35', 'port' => 26379}, {'host' => '10.0.0.9', 'port' => 26379}, {'host' => '10.0.0.6', 'port' => 26379}, ]
redis['enable'] = false
redis['master_name'] = 'redis-master' redis['master_password'] = '[REDACTED]'