Configure multiple redis instances in Kubernetes for sidekiq
Currently we are not configured for multiple redis instances which caused an incident production#2129 (closed) for the elasticsearch indexer.
gitlab-org/charts/gitlab!1287 (merged) added support in the chart to configure multiple redis and which is the version we have currently running in prod. We will need to update the config for not only sidekiq but the other redis servers.
New chart configuration
redis:
install: false
global:
redis:
host: redis.example
port: 9001
password:
enabled: true
secret: redis-secret
key: redis-password
cache:
host: cache.redis.example
port: 9002
password:
enabled: true
secret: cache-secret
key: cache-password
sharedState:
host: shared.redis.example
port: 9003
password:
enabled: true
secret: shared-secret
key: shared-password
queues:
host: queues.redis.example
port: 9004
password:
enabled: true
secret: queues-secret
key: queues-password
actioncable:
host: cable.redis.example
port: 9005
password:
enabled: true
secret: cable-secret
key: cable-password
redis:
install: false
global:
redis:
host: redis.example
sentinels:
- host: s1.redis.example
- host: s2.redis.example
Current K8s configuration
redis-cache.yml: Doesn't exist
------------------------------------------------------------------------------
redis-queues.yml: Doesn't exist
------------------------------------------------------------------------------
cable.yml:
production:
url: redis://:****@gprd-redis-sidekiq:6379
sentinels:
- host: redis-sidekiq-01-db-gprd.c.gitlab-production.internal
port: 26379
- host: redis-sidekiq-02-db-gprd.c.gitlab-production.internal
port: 26379
- host: redis-sidekiq-03-db-gprd.c.gitlab-production.internal
port: 26379
id:
adapter: redis
------------------------------------------------------------------------------
resque.yml:
production:
url: redis://:***@gprd-redis-sidekiq:6379
sentinels:
- host: redis-sidekiq-01-db-gprd.c.gitlab-production.internal
port: 26379
- host: redis-sidekiq-02-db-gprd.c.gitlab-production.internal
port: 26379
- host: redis-sidekiq-03-db-gprd.c.gitlab-production.internal
port: 26379
id:
Current VM configuration
cable.yml:
production:
adapter: redis
url: redis://:**r*@gprd-redis/
sentinels:
-
host: redis-01-db-gprd.c.gitlab-production.internal
port: 26379
-
host: redis-02-db-gprd.c.gitlab-production.internal
port: 26379
-
host: redis-03-db-gprd.c.gitlab-production.internal
port: 26379
------------------------------------------------------------------------------
redis-cache.yml:
production:
url: redis://:***@gprd-redis-cache
id:
sentinels:
-
host: redis-cache-sentinel-01-db-gprd.c.gitlab-production.internal
port: 26379
-
host: redis-cache-sentinel-02-db-gprd.c.gitlab-production.internal
port: 26379
-
host: redis-cache-sentinel-03-db-gprd.c.gitlab-production.internal
port: 26379
------------------------------------------------------------------------------
redis-queues.yml:
production:
url: redis://:***@gprd-redis-sidekiq
id:
sentinels:
-
host: redis-sidekiq-01-db-gprd.c.gitlab-production.internal
port: 26379
-
host: redis-sidekiq-02-db-gprd.c.gitlab-production.internal
port: 26379
-
host: redis-sidekiq-03-db-gprd.c.gitlab-production.internal
port: 26379
------------------------------------------------------------------------------
resque.yml:
production:
url: redis://:***@gprd-redis/
sentinels:
-
host: redis-01-db-gprd.c.gitlab-production.internal
port: 26379
-
host: redis-02-db-gprd.c.gitlab-production.internal
port: 26379
-
host: redis-03-db-gprd.c.gitlab-production.internal
port: 26379
Edited by John Jarvis