Gitlab-workhorse does not work with Redis Sentinel with TLS

Summary

Using Redis sentinel with TLS does not work with GitLab Workhorse.

The issue is that gitlab.rb does not configure Sentinel.tls in GitLab Workhorse config.toml (see below).

Steps to reproduce

Setup a host with redis + sentinel using below configuration.
Setup another host with gitlab-workhorse using below configuration.
See the logs for gitlab-workhorse and sentinel.

What is the current bug behavior?

GitLab workhorse fails to start when Redis sentinel is configured to only accept TLS traffic. GitLab fails to start.

What is the expected correct behavior?

GitLab workhorse is configured to connect to Sentinel using client certs. GitLab is functional.

Relevant logs

/var/log/gitlab/gitlab-workhorse/current

redis: 2024/12/13 11:50:45 sentinel.go:561: sentinel: GetMasterAddrByName master="gitlab-redis" failed: read: connection reset by peer

redis: 2024/12/13 11:50:45 sentinel.go:561: sentinel: GetMasterAddrByName master="gitlab-redis" failed: read: connection reset by peer

redis: 2024/12/13 11:50:46 sentinel.go:561: sentinel: GetMasterAddrByName master="gitlab-redis" failed: read: connection reset by peer

redis: 2024/12/13 11:50:47 sentinel.go:561: sentinel: GetMasterAddrByName master="gitlab-redis" failed: read: connection reset by peer

redis: 2024/12/13 11:50:48 sentinel.go:561: sentinel: GetMasterAddrByName master="gitlab-redis" failed: read: connection reset by peer

redis: 2024/12/13 11:50:48 sentinel.go:561: sentinel: GetMasterAddrByName master="gitlab-redis" failed: read: connection reset by peer

redis: 2024/12/13 11:50:49 sentinel.go:561: sentinel: GetMasterAddrByName master="gitlab-redis" failed: read: connection reset by peer

redis: 2024/12/13 11:50:50 sentinel.go:561: sentinel: GetMasterAddrByName master="gitlab-redis" failed: read: connection reset by peer

{"error":"keywatcher: redis: all sentinels specified in configuration are unreachable","level":"error","msg":"","time":"2024-12-13T11:50:53Z"} ... log lines repeat

/var/log/gitlab/sentinel/current

2024-12-13_14:20:10.05617 4711:X 13 Dec 2024 14:20:10.056 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
2024-12-13_14:20:10.05889 4711:X 13 Dec 2024 14:20:10.058 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
2024-12-13_14:20:10.05930 4711:X 13 Dec 2024 14:20:10.059 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
... log lines repeat

Details of package version

Provide the package version installation details

gitlab-ee-17.6.1-ee.0.el8.x86_64

Environment details

Operating System: RHEL 8
Installation Target, remove incorrect values:
- VM: AWS
Installation Type, remove incorrect values:
- New Installation
Is there any other software running on the machine: N/A
Is this a single or multiple node installation? multiple node installation
Resources
- CPU: 8 (workhorse), 2 (redis)
- Memory total: 16 GB (workhorse), 8 GB (redis)

Configuration details

GitLab Workhorse host /var/opt/gitlab/gitlab-workhorse/config.toml (configured by gitlab.rb)

# Below is a workaround to the issue. See gitlab.rb for example of workaround
config_command = "cat /tmp/sentinel.json"

[redis]
Password = "<redacted>"
Sentinel = ["redis://sentinel-0.gitlab.<redacted>:26379","redis://sentinel-1.gitlab.<redacted>:26379","redis://sentinel-2.gitlab.<redacted>:26379"]
SentinelMaster = "<redacted>"
SentinelPassword = ""
# Missing Sentinel.tls as seen here https://gitlab.com/gitlab-org/gitlab/-/merge_requests/162132

[image_resizer]
  max_scaler_procs = 4
  max_filesize = 250000

GitLab Workhorse host /etc/gitlab/gitlab.rb

alertmanager['enable'] = true
consul['enable'] = false
consul['monitoring_service_discovery'] = false
gitaly['enable'] = false
gitlab_exporter['enable'] = false
gitlab_kas['enable'] = false
gitlab_workhorse['enable'] = true
letsencrypt['enabled'] = false
nginx['enable'] = true
postgresql['enable'] = false
praefect['enable'] = false
prometheus['enable'] = false
puma['enable'] = true
redis['enable'] = false
sidekiq['enable'] = false

external_url 'https://gitlab.<redacted>'

gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '<redacted>'
gitlab_rails['db_password'] = '<redacted>'
postgresql['sql_user'] = '<redacted>'

nginx['ssl_certificate'] = '/opt/gitlab/ssl/nginx.pem'
nginx['ssl_certificate_key'] = '/opt/gitlab/ssl/nginx.key'
nginx['ssl_client_certificate'] = '/etc/gitlab/trusted-certs/ca.pem'

git_data_dirs({ 'default' => { 'gitaly_address' => 'tls://<redacted>', 'gitaly_token' => '<redacted>' } })

redis['master_name'] = '<redacted>'
redis['master_password'] = '<redacted>'
gitlab_rails['redis_host'] = 'sentinel-0.gitlab.<redacted>'
gitlab_rails['redis_port'] = 6379
gitlab_rails['redis_ssl'] = true
gitlab_rails['redis_tls_ca_cert_file'] = '/etc/gitlab/trusted-certs/ca.pem'
gitlab_rails['redis_tls_client_cert_file'] = '/opt/gitlab/ssl/nginx.pem'
gitlab_rails['redis_tls_client_key_file'] = '/opt/gitlab/ssl/nginx.key'
gitlab_workhorse['redis_ssl'] = true
gitlab_workhorse['redis_host'] = 'sentinel-0.gitlab.<redacted>'
gitlab_workhorse['redis_port'] = 6379
gitlab_workhorse['redis_sentinels'] = [ {'host' => 'sentinel-0.gitlab.<redacted>', 'port' => '26379'}, {'host' => 'sentinel-1.gitlab.<redacted>', 'port' => '26379'}, {'host' => 'sentinel-2.gitlab.<redacted>', 'port' => '26379'}, ]
# Workaround to the issue. Remove to reproduced. Create below file at /tpm/sentinel.json to resolve and uncomment extra_config_command
# {"Sentinel": {"tls": {"certificate": "/opt/gitlab/ssl/nginx.pem","key": "/opt/gitlab/ssl/nginx.key","ca_certificate": "/etc/gitlab/trusted-certs/ca.pem"}}}
# gitlab_workhorse['extra_config_command'] = 'cat /tmp/sentinel.json'

GitLab Redis host /etc/gitlab/gitlab.rb

alertmanager['enable'] = false
consul['enable'] = false
consul['monitoring_service_discovery'] = false
gitaly['enable'] = false
gitlab_exporter['enable'] = false
gitlab_kas['enable'] = false
gitlab_rails['auto_migrate'] = false
gitlab_workhorse['enable'] = false
letsencrypt['enabled'] = false
nginx['enable'] = false
postgresql['enable'] = false
praefect['auto_migrate'] = false
praefect['enable'] = false
prometheus['enable'] = false
puma['enable'] = false
redis['enable'] = true
sidekiq['enable'] = false

roles ['redis_master_role','redis_sentinel_role']

sentinel['bind'] = '0.0.0.0'
sentinel['password'] = '<redacted>'
sentinel['quorum'] = 2
sentinel['use_hostnames'] = true
sentinel['port'] = 0
sentinel['tls_port'] = 26379
sentinel['tls_cert_file'] = '/opt/gitlab/ssl/redis.pem'
sentinel['tls_key_file'] = '/opt/gitlab/ssl/redis.key'
sentinel['tls_ca_cert_file'] = '/etc/gitlab/trusted-certs/ca.pem'
sentinel['tls_replication'] = 'yes'

redis['announce_ip'] = 'sentinel-0.gitlab.<redacted>'
redis['bind'] = '0.0.0.0'
redis['master_ip'] = 'sentinel-0.gitlab.<redacted>'
redis['master_name'] = '<redacted>'
redis['master_password'] = '<redacted>'
redis['password'] = '<redacted>'
redis['port'] = 0
redis['tls_port'] = 6379
redis['tls_cert_file'] = '/opt/gitlab/ssl/redis.pem'
redis['tls_key_file'] = '/opt/gitlab/ssl/redis.key'
redis['tls_ca_cert_file'] = '/etc/gitlab/trusted-certs/ca.pem'
redis['tls_replication'] = 'yes'