Is there documentation about using redis dependency in sentinel mode?

Is there documentation about using redis dependency in sentinel mode?

@dm3ch Please see https://docs.gitlab.com/charts/charts/globals#redis-sentinel-support

marked this issue as related to #1810 (closed)

This might be of use: #1810 (comment 280250529)

added devopssystems documentation groupdistribution labels

@WarheadsSE It helped me, thank you a lot. I think it could be useful for other users to have #1810 (comment 280250529) example in examples directory.

Did this work with the included bitnami/redis as of 4.0.x?

Yep, but I need some additional testing before saying that it works fine

Let us know how this works out. If my comment was accurate, and this works fine with the included dependency chart, we may look to directly documenting that. Perhaps, even consider removing the explicit recommendation that the in-chart dependency not be used for production

Interesting just found that after migration test instance to sentinel instance is unavailable (but all pods are ready). And in workhorse I have next logs:

+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
Begin parsing .erb files from /var/opt/gitlab/templates
Writing /srv/gitlab/config/workhorse-config.toml
Copying other config files found in /var/opt/gitlab/templates
+ exec /bin/sh -c /scripts/start-workhorse
time="2020-06-03T10:52:05Z" level=info msg=Starting build_time=20200529.105949 version=v8.31.1
time="2020-06-03T10:52:05Z" level=info msg="redis: using sentinel" host="gitlab-stage-redis-headless:26379" scheme=tcp
time="2020-06-03T10:52:05Z" level=info msg="keywatcher: starting process loop"
time="2020-06-03T10:52:05Z" level=info msg="redis: dialing" address="127.0.0.1:6379" network=tcp
time="2020-06-03T10:52:05Z" level=error msg="unknown error" error="keywatcher: dial tcp 127.0.0.1:6379: connect: connection refused"
time="2020-06-03T10:52:06Z" level=info msg="redis: dialing" address="127.0.0.1:6379" network=tcp
time="2020-06-03T10:52:06Z" level=error msg="unknown error" error="keywatcher: dial tcp 127.0.0.1:6379: connect: connection refused"

@WarheadsSE I had found that redis connectivity problems was caused by current verion of redis dependency chart.

On current version I got a problem that sentinel have anounced master as 127.0.0.1 . Upgrading dependency to latest version (10.6.18) seems helped me.

@WarheadsSE It seems that it's still not ready for production usage.

Suddenly redis chart in sentinel configuration not performs failover before actual master pod shutdown. So even if you are gracefully stoping actual master pod you still needs to wait downAfterMilliseconds timeout. :(

https://github.com/bitnami/charts/issues/2749

Can we move this discussion / work to #1810 (closed)?

Hmm i'm not actually sure. It seems that #1810 (closed) is not actual, cause gitlab could work with redis sentinel installed by chart dependency (with new verison of redis subchart).

So there're two problems:

1. Sentinel 127.0.0.1 master sometimes. It seems that could be solved by using latest version of redis chart.
2. Master unavailability after gracefull shutdown of current master pod - trying to solve in https://github.com/bitnami/charts/issues/2749

So it's up to you, for me it doesn't actually matter in which issue we will continue discussion.

closed

mentioned in issue gitlab-org/distribution/team-tasks#663 (closed)