redis-ha fails to establish cluster quorum
Summary
redis-ha fails to reach cluster stability
Steps to reproduce
helm upgrade --install gitlab stable/gitlab
Configuration used
redis-ha:
nameOverride: redis
enable: true
Current behavior
replicas 1 and 2 end up CrashLoopBackOff
Expected behavior
stable cluster
Versions
-
Chart: 5122f20c
-
Platform:
- Cloud: AWS
-
Kubernetes: (
kubectl version
)
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.8", GitCommit:"7eab6a49736cc7b01869a15f9f05dc5b49efb9fc", GitTreeState:"clean", BuildDate:"2018-09-14T15:54:20Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
- Helm: (
helm version
)
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Relevant logs
timeout: invalid option -- 't'
Try 'timeout --help' for more information.
+ [[ 125 == \0 ]]
+ i=24
+ [[ 24 -gt 30 ]]
+ echo 'Connecting to master failed. Waiting...'
+ sleep 1
Connecting to master failed. Waiting...
Could not connect to Redis at 127.0.0.1:6379: Connection refused
Could not connect to Redis at 127.0.0.1:6379: Connection refused
Connecting to master failed. Waiting...
+ true
+ master=100.68.84.146
+ timeout -t 3 redis-cli -a E0S2YyKokyplcfj70wpV78A9cXHYvTRZjM3Vanhh2OWHcQG835iVfJ2MPFO9h7Hx -h 100.68.84.146 -p 6379 INFO
timeout: invalid option -- 't'
Try 'timeout --help' for more information.
+ [[ 125 == \0 ]]
+ i=25
+ [[ 25 -gt 30 ]]
+ echo 'Connecting to master failed. Waiting...'
+ sleep 1
Could not connect to Redis at 127.0.0.1:6379: Connection refused
Could not connect to Redis at 127.0.0.1:6379: Connection refused
+ true
+ master=100.68.84.146
+ timeout -t 3 redis-cli -a E0S2YyKokyplcfj70wpV78A9cXHYvTRZjM3Vanhh2OWHcQG835iVfJ2MPFO9h7Hx -h 100.68.84.146 -p 6379 INFO
timeout: invalid option -- 't'
Try 'timeout --help' for more information.
+ [[ 125 == \0 ]]
+ i=26
+ [[ 26 -gt 30 ]]
+ echo 'Connecting to master failed. Waiting...'
+ sleep 1
Connecting to master failed. Waiting...
Could not connect to Redis at 127.0.0.1:6379: Connection refused
Could not connect to Redis at 127.0.0.1:6379: Connection refused
Connecting to master failed. Waiting...
+ true
+ master=100.68.84.146
+ timeout -t 3 redis-cli -a E0S2YyKokyplcfj70wpV78A9cXHYvTRZjM3Vanhh2OWHcQG835iVfJ2MPFO9h7Hx -h 100.68.84.146 -p 6379 INFO
timeout: invalid option -- 't'
Try 'timeout --help' for more information.
+ [[ 125 == \0 ]]
+ i=27
+ [[ 27 -gt 30 ]]
+ echo 'Connecting to master failed. Waiting...'
+ sleep 1
Could not connect to Redis at 127.0.0.1:6379: Connection refused
Could not connect to Redis at 127.0.0.1:6379: Connection refused
+ true
+ master=100.68.84.146
+ timeout -t 3 redis-cli -a E0S2YyKokyplcfj70wpV78A9cXHYvTRZjM3Vanhh2OWHcQG835iVfJ2MPFO9h7Hx -h 100.68.84.146 -p 6379 INFO
timeout: invalid option -- 't'
Try 'timeout --help' for more information.
Connecting to master failed. Waiting...
+ [[ 125 == \0 ]]
+ i=28
+ [[ 28 -gt 30 ]]
+ echo 'Connecting to master failed. Waiting...'
+ sleep 1
Could not connect to Redis at 127.0.0.1:6379: Connection refused
Could not connect to Redis at 127.0.0.1:6379: Connection refused
+ true
+ master=100.68.84.146
+ timeout -t 3 redis-cli -a E0S2YyKokyplcfj70wpV78A9cXHYvTRZjM3Vanhh2OWHcQG835iVfJ2MPFO9h7Hx -h 100.68.84.146 -p 6379 INFO
timeout: invalid option -- 't'
Try 'timeout --help' for more information.
+ [[ 125 == \0 ]]
+ i=29
+ [[ 29 -gt 30 ]]
+ echo 'Connecting to master failed. Waiting...'
+ sleep 1
Connecting to master failed. Waiting...
Could not connect to Redis at 127.0.0.1:6379: Connection refused
Could not connect to Redis at 127.0.0.1:6379: Connection refused
+ true
+ master=100.68.84.146
+ timeout -t 3 redis-cli -a E0S2YyKokyplcfj70wpV78A9cXHYvTRZjM3Vanhh2OWHcQG835iVfJ2MPFO9h7Hx -h 100.68.84.146 -p 6379 INFO
timeout: invalid option -- 't'
Try 'timeout --help' for more information.
+ [[ 125 == \0 ]]
+ i=30
+ [[ 30 -gt 30 ]]
+ echo 'Connecting to master failed. Waiting...'
+ sleep 1
Connecting to master failed. Waiting...
Could not connect to Redis at 127.0.0.1:6379: Connection refused
Could not connect to Redis at 127.0.0.1:6379: Connection refused
Exiting after too many attempts
+ true
+ master=100.68.84.146
+ timeout -t 3 redis-cli -a E0S2YyKokyplcfj70wpV78A9cXHYvTRZjM3Vanhh2OWHcQG835iVfJ2MPFO9h7Hx -h 100.68.84.146 -p 6379 INFO
timeout: invalid option -- 't'
Try 'timeout --help' for more information.
+ [[ 125 == \0 ]]
+ i=31
+ [[ 31 -gt 30 ]]
+ echo 'Exiting after too many attempts'
+ kill 9
+ exit 1
Edited by Jason Plum