Reconfigure on sentinel node fails if only tls is available
Summary
The second reconfigure will fail on a gitlab omnibus sentinel node if the regular sentinel port is disabled, and only the tls port is enabled.
Details
There was an error running gitlab-ctl reconfigure:
sentinel_service[redis] (gitlab-ee::sentinel line 21) had an error: RuntimeError: ruby_block[warn pending sentinel restart] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/resources/sentinel_service.rb line 69) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/redis-cli -h 0.0.0.0 -p 0 INFO` failed with a non-zero exit code (1)
stdout:
stderr: Could not connect to Redis at 0.0.0.0:0: Connection refused
This was found during testing of: !5770 (merged)
The workaround for now is to keep a non-tls port available for sentinel.
This fails on the second reconfigure, as the failing piece of code isn't run during first boostrap.
Note that our redis code doesn't run into this issue because it uses the redis socket if the regular port is 0
.
Config:
roles ['redis_sentinel_role']
# Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis'
# The same password for Redis authentication you set up for the master node.
redis['master_password'] = 'your password'
redis['master_ip'] = 'master ip'
sentinel['port'] = 0
sentinel['tls_port'] = 26379
sentinel['tls_cert_file'] = '/etc/gitlab/ssl/ip-redis.crt'
sentinel['tls_key_file'] = '/etc/gitlab/ssl/ip-redis.key'
sentinel['tls_replication'] = 'yes'
sentinel['bind'] = '0.0.0.0'
gitlab_rails['auto_migrate'] = false
sentinel['quorum'] = 2
Proposed fixes
Add support for using the tls port in sentinel helper when checking the running version, and also add the --tls
flag to the redis-cli command that is used there. https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/f90d17a0656a38aeeb07e63ac624c2a1909153f0/files/gitlab-cookbooks/gitlab-ee/libraries/sentinel_helper.rb#L20