Websocket servers throwing 502's - not receiving new messages

Gitter websocket servers throwing 502's.

We have some initial alerts about the servers throwing a 502 when trying to check for the SSL cert:

• https://gitter.pagerduty.com/incidents/PRNJL9Y

Then it looks like some of WS processes are recycled as they spiral out of CPU and memory,

• https://gitter.pagerduty.com/incidents/PP9ARDI
• https://gitter.pagerduty.com/incidents/P1AB5HB

From the ws-servers ASG, I only see one server that got recycled, ws-04 went down and ws-09 spun up,

Redis

I wonder if this has anything to do with the Redis sentinel bump, redis_sentinel_sdown. We use Redis to track Faye/bayeux subscriptions. Things started to bubble up at 15:22UTC in the logs which aligns with all of the datadog stuff

• https://gitter.pagerduty.com/incidents/PPCXSQR
• https://gitter.pagerduty.com/incidents/P1JSVS8

Also mentioned in https://gitlab.com/gitlab-org/gitter/webapp/issues/2382#note_258301892

sentinel-01 redis-sentinel.log.1

$ ssh sentinel-01.prod.gitter
$ cat /var/log/upstart/redis-sentinel.log.1 | grep "22110:X 10 Dec"

22110:X 10 Dec 15:22:40.990 # +sdown master gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:22:41.090 # +odown master gitter-master 10.0.1.137 6379 #quorum 2/2
22110:X 10 Dec 15:22:41.091 # +new-epoch 96
22110:X 10 Dec 15:22:41.091 # +try-failover master gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:22:41.094 # +vote-for-leader d020820076b08590d8a9a6b7edd3be0579e07ba6 96
22110:X 10 Dec 15:22:41.116 # 6b42b1c8134a41707653abb24efda700dec851a4 voted for 6b42b1c8134a41707653abb24efda700dec851a4 96
22110:X 10 Dec 15:22:41.134 # 6045a11ab1f90e1dc41be87f6c0de95dbbb24854 voted for 6b42b1c8134a41707653abb24efda700dec851a4 96
22110:X 10 Dec 15:22:42.240 # +config-update-from sentinel 6b42b1c8134a41707653abb24efda700dec851a4 10.0.21.133 26379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:22:42.241 # +switch-master gitter-master 10.0.1.137 6379 10.0.11.124 6379
22110:X 10 Dec 15:22:42.242 * +slave slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
22110:X 10 Dec 15:23:00.276 # +sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
22110:X 10 Dec 15:23:05.343 # -sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
22110:X 10 Dec 15:23:12.833 # +sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
22110:X 10 Dec 15:23:29.102 # -sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
22110:X 10 Dec 15:27:57.755 # +sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
22110:X 10 Dec 15:28:56.552 # -sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
22110:X 10 Dec 15:41:51.074 # +sdown master gitter-master 10.0.11.124 6379
22110:X 10 Dec 15:41:51.106 # +new-epoch 97
22110:X 10 Dec 15:41:51.107 # +vote-for-leader 6b42b1c8134a41707653abb24efda700dec851a4 97
22110:X 10 Dec 15:41:51.133 # +odown master gitter-master 10.0.11.124 6379 #quorum 3/2
22110:X 10 Dec 15:41:51.133 # Next failover delay: I will not start a failover before Tue Dec 10 15:42:52 2019
22110:X 10 Dec 15:41:52.272 # +config-update-from sentinel 6b42b1c8134a41707653abb24efda700dec851a4 10.0.21.133 26379 @ gitter-master 10.0.11.124 6379
22110:X 10 Dec 15:41:52.273 # +switch-master gitter-master 10.0.11.124 6379 10.0.1.137 6379
22110:X 10 Dec 15:41:52.273 * +slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:08.494 # +sdown master gitter-faye-prod 10.0.11.124 6382
22110:X 10 Dec 15:42:08.586 # +odown master gitter-faye-prod 10.0.11.124 6382 #quorum 3/2
22110:X 10 Dec 15:42:08.586 # +new-epoch 98
22110:X 10 Dec 15:42:08.587 # +try-failover master gitter-faye-prod 10.0.11.124 6382
22110:X 10 Dec 15:42:08.589 # +vote-for-leader d020820076b08590d8a9a6b7edd3be0579e07ba6 98
22110:X 10 Dec 15:42:08.615 # 6b42b1c8134a41707653abb24efda700dec851a4 voted for d020820076b08590d8a9a6b7edd3be0579e07ba6 98
22110:X 10 Dec 15:42:08.646 # 6045a11ab1f90e1dc41be87f6c0de95dbbb24854 voted for 6045a11ab1f90e1dc41be87f6c0de95dbbb24854 98
22110:X 10 Dec 15:42:08.951 # +sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:12.550 # -sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:15.939 # -sdown master gitter-faye-prod 10.0.11.124 6382
22110:X 10 Dec 15:42:15.940 # -odown master gitter-faye-prod 10.0.11.124 6382
22110:X 10 Dec 15:42:18.864 # -failover-abort-not-elected master gitter-faye-prod 10.0.11.124 6382
22110:X 10 Dec 15:42:20.216 # +sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:25.799 # -sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:33.949 # +sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.982 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.982 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.983 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.983 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.983 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.984 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.984 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.986 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.987 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.987 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.987 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.987 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.988 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.988 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.988 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.990 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.990 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.991 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.991 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.991 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.991 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.992 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.993 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.994 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.994 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.994 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.995 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.995 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.995 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.995 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.997 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.997 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.997 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.998 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:46.998 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:47.024 # -sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.696 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.697 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.697 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.697 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.698 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.698 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.698 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.701 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.701 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.701 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.701 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.702 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.702 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.702 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.703 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.704 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.704 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.705 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.705 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.705 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.706 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.706 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.707 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.708 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.708 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.708 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.709 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.709 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.709 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.709 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.711 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.711 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.711 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.712 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.712 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.712 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.712 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.713 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.714 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.714 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.715 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.715 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.715 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.715 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.716 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.717 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.717 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.718 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.718 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.718 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.718 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.719 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.719 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.720 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.721 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.721 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.721 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.721 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.722 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.722 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.722 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:42:50.724 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:47:14.481 # +sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
22110:X 10 Dec 15:47:59.321 # -sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379

sentinel-02 redis-sentinel.log.1

$ ssh sentinel-02.prod.gitter
$ cat /var/log/upstart/redis-sentinel.log.1 | grep "31670:X 10 Dec"

31670:X 10 Dec 15:22:52.407 # +sdown master gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:22:53.193 # +new-epoch 96
31670:X 10 Dec 15:22:53.196 # +vote-for-leader 6b42b1c8134a41707653abb24efda700dec851a4 96
31670:X 10 Dec 15:22:53.569 # +odown master gitter-master 10.0.1.137 6379 #quorum 3/2
31670:X 10 Dec 15:22:53.570 # Next failover delay: I will not start a failover before Tue Dec 10 15:23:53 2019
31670:X 10 Dec 15:22:54.329 # +config-update-from sentinel 6b42b1c8134a41707653abb24efda700dec851a4 10.0.21.133 26379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:22:54.329 # +switch-master gitter-master 10.0.1.137 6379 10.0.11.124 6379
31670:X 10 Dec 15:22:54.330 * +slave slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
31670:X 10 Dec 15:23:12.305 # +sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
31670:X 10 Dec 15:23:17.402 # -sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
31670:X 10 Dec 15:23:18.141 # +sdown slave 10.0.1.137:6382 10.0.1.137 6382 @ gitter-faye-prod 10.0.11.124 6382
31670:X 10 Dec 15:23:18.817 # -sdown slave 10.0.1.137:6382 10.0.1.137 6382 @ gitter-faye-prod 10.0.11.124 6382
31670:X 10 Dec 15:23:24.946 # +sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
31670:X 10 Dec 15:23:38.911 * +convert-to-slave slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
31670:X 10 Dec 15:23:39.843 # -sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
31670:X 10 Dec 15:28:10.543 # +sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
31670:X 10 Dec 15:29:08.690 # -sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
31670:X 10 Dec 15:42:02.964 # +sdown master gitter-master 10.0.11.124 6379
31670:X 10 Dec 15:42:03.195 # +new-epoch 97
31670:X 10 Dec 15:42:03.198 # +vote-for-leader 6b42b1c8134a41707653abb24efda700dec851a4 97
31670:X 10 Dec 15:42:04.103 # +odown master gitter-master 10.0.11.124 6379 #quorum 3/2
31670:X 10 Dec 15:42:04.104 # Next failover delay: I will not start a failover before Tue Dec 10 15:43:03 2019
31670:X 10 Dec 15:42:04.362 # +config-update-from sentinel 6b42b1c8134a41707653abb24efda700dec851a4 10.0.21.133 26379 @ gitter-master 10.0.11.124 6379
31670:X 10 Dec 15:42:04.363 # +switch-master gitter-master 10.0.11.124 6379 10.0.1.137 6379
31670:X 10 Dec 15:42:04.363 * +slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:20.601 # +sdown master gitter-faye-prod 10.0.11.124 6382
31670:X 10 Dec 15:42:20.686 # +odown master gitter-faye-prod 10.0.11.124 6382 #quorum 3/2
31670:X 10 Dec 15:42:20.687 # +new-epoch 98
31670:X 10 Dec 15:42:20.687 # +try-failover master gitter-faye-prod 10.0.11.124 6382
31670:X 10 Dec 15:42:20.690 # +vote-for-leader 6045a11ab1f90e1dc41be87f6c0de95dbbb24854 98
31670:X 10 Dec 15:42:20.738 # d020820076b08590d8a9a6b7edd3be0579e07ba6 voted for d020820076b08590d8a9a6b7edd3be0579e07ba6 98
31670:X 10 Dec 15:42:20.738 # 6b42b1c8134a41707653abb24efda700dec851a4 voted for d020820076b08590d8a9a6b7edd3be0579e07ba6 98
31670:X 10 Dec 15:42:21.087 # +sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:24.670 # -sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:28.023 # -sdown master gitter-faye-prod 10.0.11.124 6382
31670:X 10 Dec 15:42:28.023 # -odown master gitter-faye-prod 10.0.11.124 6382
31670:X 10 Dec 15:42:31.386 # -failover-abort-not-elected master gitter-faye-prod 10.0.11.124 6382
31670:X 10 Dec 15:42:32.296 # +sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:37.919 # -sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:46.077 # +sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.072 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.072 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.073 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.073 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.074 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.074 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.074 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.077 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.077 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.077 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.078 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.078 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.078 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.079 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.079 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.081 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.081 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.081 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.082 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.082 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.082 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.082 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.084 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.084 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.085 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.085 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.085 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.086 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.086 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.086 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.088 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.088 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.088 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.088 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.089 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:42:59.159 # -sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:47:26.554 # +sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
31670:X 10 Dec 15:48:11.455 # -sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379

sentinel-03 redis-sentinel.log.1

$ ssh sentinel-03.prod.gitter
$ cat /var/log/upstart/redis-sentinel.log.1 | grep "22110:X 10 Dec"

cat /var/log/upstart/redis-sentinel.log.1 | grep "11714:X 10 Dec"
11714:X 10 Dec 15:22:22.498 # +sdown master gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:22:22.557 # +odown master gitter-master 10.0.1.137 6379 #quorum 3/2
11714:X 10 Dec 15:22:22.557 # +new-epoch 96
11714:X 10 Dec 15:22:22.557 # +try-failover master gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:22:22.560 # +vote-for-leader 6b42b1c8134a41707653abb24efda700dec851a4 96
11714:X 10 Dec 15:22:22.572 # 6045a11ab1f90e1dc41be87f6c0de95dbbb24854 voted for 6b42b1c8134a41707653abb24efda700dec851a4 96
11714:X 10 Dec 15:22:22.579 # d020820076b08590d8a9a6b7edd3be0579e07ba6 voted for d020820076b08590d8a9a6b7edd3be0579e07ba6 96
11714:X 10 Dec 15:22:22.615 # +elected-leader master gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:22:22.615 # +failover-state-select-slave master gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:22:22.689 # +selected-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:22:22.689 * +failover-state-send-slaveof-noone slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:22:22.766 * +failover-state-wait-promotion slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:22:23.627 # +promoted-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:22:23.628 # +failover-state-reconf-slaves master gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:22:23.697 # +failover-end master gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:22:23.697 # +switch-master gitter-master 10.0.1.137 6379 10.0.11.124 6379
11714:X 10 Dec 15:22:23.697 * +slave slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:22:41.699 # +sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:22:46.780 # -sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:22:47.556 # +sdown slave 10.0.1.137:6382 10.0.1.137 6382 @ gitter-faye-prod 10.0.11.124 6382
11714:X 10 Dec 15:22:48.244 # -sdown slave 10.0.1.137:6382 10.0.1.137 6382 @ gitter-faye-prod 10.0.11.124 6382
11714:X 10 Dec 15:22:54.256 # +sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:23:08.311 * +convert-to-slave slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:23:10.807 # -sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:27:39.895 # +sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:28:38.066 # -sdown slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:41:32.501 # +sdown master gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:41:32.561 # +odown master gitter-master 10.0.11.124 6379 #quorum 2/2
11714:X 10 Dec 15:41:32.561 # +new-epoch 97
11714:X 10 Dec 15:41:32.561 # +try-failover master gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:41:32.563 # +vote-for-leader 6b42b1c8134a41707653abb24efda700dec851a4 97
11714:X 10 Dec 15:41:32.572 # d020820076b08590d8a9a6b7edd3be0579e07ba6 voted for 6b42b1c8134a41707653abb24efda700dec851a4 97
11714:X 10 Dec 15:41:32.573 # 6045a11ab1f90e1dc41be87f6c0de95dbbb24854 voted for 6b42b1c8134a41707653abb24efda700dec851a4 97
11714:X 10 Dec 15:41:32.626 # +elected-leader master gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:41:32.626 # +failover-state-select-slave master gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:41:32.686 # +selected-slave slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:41:32.686 * +failover-state-send-slaveof-noone slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:41:32.745 * +failover-state-wait-promotion slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:41:33.632 # +promoted-slave slave 10.0.1.137:6379 10.0.1.137 6379 @ gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:41:33.632 # +failover-state-reconf-slaves master gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:41:33.729 # +failover-end master gitter-master 10.0.11.124 6379
11714:X 10 Dec 15:41:33.729 # +switch-master gitter-master 10.0.11.124 6379 10.0.1.137 6379
11714:X 10 Dec 15:41:33.729 * +slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:41:45.774 # +sdown master gitter-faye-prod 10.0.11.124 6382
11714:X 10 Dec 15:41:50.078 # +new-epoch 98
11714:X 10 Dec 15:41:50.078 # +vote-for-leader d020820076b08590d8a9a6b7edd3be0579e07ba6 98
11714:X 10 Dec 15:41:50.938 # +odown master gitter-faye-prod 10.0.11.124 6382 #quorum 3/2
11714:X 10 Dec 15:41:50.938 # Next failover delay: I will not start a failover before Tue Dec 10 15:42:50 2019
11714:X 10 Dec 15:41:51.237 # +sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:41:51.954 # -sdown master gitter-faye-prod 10.0.11.124 6382
11714:X 10 Dec 15:41:51.954 # -odown master gitter-faye-prod 10.0.11.124 6382
11714:X 10 Dec 15:41:54.775 # -sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:00.798 # +sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:07.256 # -sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:15.433 # +sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.446 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.446 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.446 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.446 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.446 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.446 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.446 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.446 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.446 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.446 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.447 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.447 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.447 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.447 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.447 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.447 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.447 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.447 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.447 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.447 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.447 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.448 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.449 * +convert-to-slave slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:42:28.530 # -sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:46:55.417 # +sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379
11714:X 10 Dec 15:47:40.804 # -sdown slave 10.0.11.124:6379 10.0.11.124 6379 @ gitter-master 10.0.1.137 6379

WebSockets and Faye

https://app.datadoghq.com/dashboard/spf-vjw-ujs/websocket-servers?from_ts=1575837556116&live=true&tile_size=m&to_ts=1576010356116

https://app.datadoghq.com/dashboard/g9p-cpa-fd8/faye?from_ts=1575837558109&live=true&tile_size=m&to_ts=1576010358109

Cause

TBD

Reference on a different Redis related failure we had before: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4143

Mitigated

Wait for Redis and ws-xx servers to settle down

Restart webapp-xx and ws-xx servers. I assume this fixed the Redis connections after the Redis sentinel changed the masters/slaves

Fix

TBD