2017-05-14 ~11:44 Redis failover
It looks like around 11:44 UTC time today we had a Redis master failover from redis6 to redis4. Then at 12:11 Redis switched masters from redis4 to redis6:
I don't see any host changes for redis6 that might have caused this:
It does look like GitLab was down for 5 minutes while slaves were attempting to do a full resync.
redis7 logs show:
```
2017-05-14_11:42:17.01414 redis7 redis: 8576:S 14 May 11:42:17.014 * 10000 changes in 60 seconds. Saving...
2017-05-14_11:42:18.04146 redis7 redis: 8576:S 14 May 11:42:18.041 * Background saving started by pid 12134
2017-05-14_11:44:58.69898 redis7 redis: 8576:S 14 May 11:44:58.698 # Connection with master lost.
2017-05-14_11:44:58.69904 redis7 redis: 8576:S 14 May 11:44:58.698 * Caching the disconnected master state.
2017-05-14_11:44:58.69905 redis7 redis: 8576:S 14 May 11:44:58.698 * Discarding previously cached master state.
2017-05-14_11:44:58.69907 redis7 redis: 8576:S 14 May 11:44:58.698 * SLAVE OF 10.45.2.7:6379 enabled (user request from 'id=197667 addr=10.45.2.8:60641 fd=12 name=sentinel-734d44a1-cmd age=60 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=0 qbuf-free=32768 obl=36 oll=0 omem=0 events=r cmd=exec')
2017-05-14_11:44:58.77238 redis7 redis: 8576:S 14 May 11:44:58.772 # CONFIG REWRITE executed with success.
2017-05-14_11:44:59.07326 redis7 redis: 8576:S 14 May 11:44:59.073 * Connecting to MASTER 10.45.2.7:6379
2017-05-14_11:44:59.07331 redis7 redis: 8576:S 14 May 11:44:59.073 * MASTER <-> SLAVE sync started
2017-05-14_11:44:59.07377 redis7 redis: 8576:S 14 May 11:44:59.073 * Non blocking connect for SYNC fired the event.
2017-05-14_11:44:59.07399 redis7 redis: 8576:S 14 May 11:44:59.073 * Master replied to PING, replication can continue...
2017-05-14_11:44:59.07458 redis7 redis: 8576:S 14 May 11:44:59.074 * Partial resynchronization not possible (no cached master)
2017-05-14_11:47:13.17900 redis7 redis: 12134:C 14 May 11:47:13.178 * DB saved on disk
2017-05-14_11:47:13.97429 redis7 redis: 12134:C 14 May 11:47:13.974 * RDB: 223 MB of memory used by copy-on-write
2017-05-14_11:47:15.07335 redis7 redis: 8576:S 14 May 11:47:15.073 * Background saving terminated with success
2017-05-14_11:48:03.44092 redis7 redis: 8576:S 14 May 11:48:03.440 * Full resync from master: 3501a3950341b7b7f076e2aeb15a97b60470ad52:127640466534
2017-05-14_11:48:16.07424 redis7 redis: 8576:S 14 May 11:48:16.074 * 10000 changes in 60 seconds. Saving...
2017-05-14_11:48:17.09327 redis7 redis: 8576:S 14 May 11:48:17.093 * Background saving started by pid 12591
2017-05-14_11:53:17.06233 redis7 redis: 12591:C 14 May 11:53:17.062 * DB saved on disk
2017-05-14_11:53:17.89856 redis7 redis: 12591:C 14 May 11:53:17.898 * RDB: 10 MB of memory used by copy-on-write
2017-05-14_11:53:19.01338 redis7 redis: 8576:S 14 May 11:53:19.013 * Background saving terminated with success
2017-05-14_11:53:39.76748 redis7 redis: 8576:S 14 May 11:53:39.767 * MASTER <-> SLAVE sync: receiving 17637910545 bytes from master
2017-05-14_12:01:21.53339 redis7 redis: 8576:S 14 May 12:01:21.533 * MASTER <-> SLAVE sync: Flushing old data
2017-05-14_12:05:24.33912 redis7 redis: 8576:S 14 May 12:05:24.339 * MASTER <-> SLAVE sync: Loading DB in memory
2017-05-14_12:11:44.87722 redis7 redis: 8576:S 14 May 12:11:44.877 * MASTER <-> SLAVE sync: Finished with success
2017-05-14_12:11:44.97911 redis7 redis: 8576:S 14 May 12:11:44.979 * 1 changes in 900 seconds. Saving...
2017-05-14_12:11:45.59587 redis7 redis: 8576:S 14 May 12:11:45.595 * Background saving started by pid 14843
2017-05-14_12:11:50.82575 redis7 redis: 8576:S 14 May 12:11:50.825 # Connection with master lost.
2017-05-14_12:11:50.82584 redis7 redis: 8576:S 14 May 12:11:50.825 * Caching the disconnected master state.
2017-05-14_12:11:50.82591 redis7 redis: 8576:S 14 May 12:11:50.825 * Discarding previously cached master state.
2017-05-14_12:11:50.82600 redis7 redis: 8576:S 14 May 12:11:50.825 * SLAVE OF 10.45.2.9:6379 enabled (user request from 'id=198404 addr=10.45.2.8:43206 fd=337 name= age=386 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=14 qbuf-free=32754 obl=36 oll=0 omem=0 events=r cmd=exec')
2017-05-14_12:11:50.83199 redis7 redis: 8576:S 14 May 12:11:50.831 # CONFIG REWRITE executed with success.
2017-05-14_12:11:51.10123 redis7 redis: 8576:S 14 May 12:11:51.101 * Connecting to MASTER 10.45.2.9:6379
2017-05-14_12:11:51.10134 redis7 redis: 8576:S 14 May 12:11:51.101 * MASTER <-> SLAVE sync started
2017-05-14_12:11:51.10179 redis7 redis: 8576:S 14 May 12:11:51.101 * Non blocking connect for SYNC fired the event.
2017-05-14_12:11:51.10224 redis7 redis: 8576:S 14 May 12:11:51.102 * Master replied to PING, replication can continue...
2017-05-14_12:11:51.10300 redis7 redis: 8576:S 14 May 12:11:51.102 * Partial resynchronization not possible (no cached master)
2017-05-14_12:17:20.18978 redis7 redis: 14843:C 14 May 12:17:20.189 * DB saved on disk
2017-05-14_12:17:20.66431 redis7 redis: 14843:C 14 May 12:17:20.664 * RDB: 418 MB of memory used by copy-on-write
2017-05-14_12:17:21.30776 redis7 redis: 8576:S 14 May 12:17:21.307 * Background saving terminated with success
2017-05-14_12:17:46.23719 redis7 redis: 8576:S 14 May 12:17:46.237 * Full resync from master: 405b30be07f1a319117af61bbb7ceb46665f0155:127939518653
2017-05-14_12:18:22.02955 redis7 redis: 8576:S 14 May 12:18:22.029 * 10000 changes in 60 seconds. Saving...
2017-05-14_12:18:22.62514 redis7 redis: 8576:S 14 May 12:18:22.625 * Background saving started by pid 16982
2017-05-14_12:23:54.41540 redis7 redis: 16982:C 14 May 12:23:54.415 * DB saved on disk
2017-05-14_12:23:54.89169 redis7 redis: 16982:C 14 May 12:23:54.891 * RDB: 7 MB of memory used by copy-on-write
2017-05-14_12:23:55.62107 redis7 redis: 8576:S 14 May 12:23:55.621 * Background saving terminated with success
2017-05-14_12:24:02.85448 redis7 redis: 8576:S 14 May 12:24:02.854 * MASTER <-> SLAVE sync: receiving 17638136249 bytes from master
2017-05-14_12:31:22.42579 redis7 redis: 8576:S 14 May 12:31:22.425 * MASTER <-> SLAVE sync: Flushing old data
2017-05-14_12:34:17.86499 redis7 redis: 8576:S 14 May 12:34:17.864 * MASTER <-> SLAVE sync: Loading DB in memory
2017-05-14_12:38:16.77961 redis7 redis: 8576:S 14 May 12:38:16.779 * MASTER <-> SLAVE sync: Finished with success
```
redis6 logs show:
```
2017-05-14_11:35:46.55560 redis6 redis: 51777:C 14 May 11:35:46.555 * DB saved on disk
2017-05-14_11:35:47.37226 redis6 redis: 51777:C 14 May 11:35:47.372 * RDB: 1015 MB of memory used by copy-on-write
2017-05-14_11:35:48.60070 redis6 redis: 54900:M 14 May 11:35:48.600 * Background saving terminated with success
2017-05-14_11:36:49.06570 redis6 redis: 54900:M 14 May 11:36:49.065 * 10000 changes in 60 seconds. Saving...
2017-05-14_11:36:50.04040 redis6 redis: 54900:M 14 May 11:36:50.040 * Background saving started by pid 52305
2017-05-14_11:43:46.64622 redis6 redis: 54900:M 14 May 11:43:46.646 # Background saving terminated by signal 9
2017-05-14_11:43:47.17303 redis6 redis: 54900:M 14 May 11:43:47.172 * 10 changes in 300 seconds. Saving...
2017-05-14_11:43:48.25121 redis6 redis: 54900:M 14 May 11:43:48.251 * Background saving started by pid 54488
2017-05-14_11:44:18.56448 redis6 redis: 54900:S 14 May 11:44:18.564 # Connection with slave 10.45.2.6:6379 lost.
2017-05-14_11:44:18.56473 redis6 redis: 54900:S 14 May 11:44:18.564 # Connection with slave 10.45.2.10:6379 lost.
2017-05-14_11:44:18.56487 redis6 redis: 54900:S 14 May 11:44:18.564 # Connection with slave 10.45.2.8:6379 lost.
2017-05-14_11:44:18.56513 redis6 redis: 54900:S 14 May 11:44:18.565 # Connection with slave 10.45.2.7:6379 lost.
2017-05-14_11:44:18.56535 redis6 redis: 54900:S 14 May 11:44:18.565 * SLAVE OF 10.45.2.7:6379 enabled (user request from 'id=10024214 addr=10.45.2.7:36116 fd=321 name=sentinel-53dd2433-cmd age=20 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=0 qbuf-free=32768 obl=36 oll=0 omem=0 events=r cmd=exec')
2017-05-14_11:44:18.57227 redis6 redis: 54900:S 14 May 11:44:18.572 # CONFIG REWRITE executed with success.
2017-05-14_11:44:18.93704 redis6 redis: 54900:S 14 May 11:44:18.936 * Connecting to MASTER 10.45.2.7:6379
2017-05-14_11:44:18.93716 redis6 redis: 54900:S 14 May 11:44:18.937 * MASTER <-> SLAVE sync started
2017-05-14_11:44:18.93784 redis6 redis: 54900:S 14 May 11:44:18.937 * Non blocking connect for SYNC fired the event.
2017-05-14_11:44:18.93813 redis6 redis: 54900:S 14 May 11:44:18.938 * Master replied to PING, replication can continue...
2017-05-14_11:44:18.93867 redis6 redis: 54900:S 14 May 11:44:18.938 * Partial resynchronization not possible (no cached master)
2017-05-14_11:48:03.44192 redis6 redis: 54900:S 14 May 11:48:03.441 * Full resync from master: 3501a3950341b7b7f076e2aeb15a97b60470ad52:127640466534
2017-05-14_11:49:00.31702 redis6 redis: 54488:C 14 May 11:49:00.316 * DB saved on disk
2017-05-14_11:49:01.09140 redis6 redis: 54488:C 14 May 11:49:01.091 * RDB: 191 MB of memory used by copy-on-write
2017-05-14_11:49:02.09385 redis6 redis: 54900:S 14 May 11:49:02.093 * Background saving terminated with success
2017-05-14_11:53:39.76867 redis6 redis: 54900:S 14 May 11:53:39.768 * MASTER <-> SLAVE sync: receiving 17637910545 bytes from master
2017-05-14_12:01:21.43507 redis6 redis: 54900:S 14 May 12:01:21.435 * MASTER <-> SLAVE sync: Flushing old data
2017-05-14_12:05:15.10493 redis6 redis: 54900:S 14 May 12:05:15.104 * MASTER <-> SLAVE sync: Loading DB in memory
2017-05-14_12:11:32.55637 redis6 redis: 54900:S 14 May 12:11:32.556 * MASTER <-> SLAVE sync: Finished with success
2017-05-14_12:11:32.55644 redis6 redis: 54900:S 14 May 12:11:32.556 * 1 changes in 900 seconds. Saving...
2017-05-14_12:11:33.13982 redis6 redis: 54900:S 14 May 12:11:33.139 * Background saving started by pid 56832
2017-05-14_12:11:49.90537 redis6 redis: 54900:M 14 May 12:11:49.905 # Connection with master lost.
2017-05-14_12:11:49.90552 redis6 redis: 54900:M 14 May 12:11:49.905 * Caching the disconnected master state.
2017-05-14_12:11:49.90561 redis6 redis: 54900:M 14 May 12:11:49.905 * Discarding previously cached master state.
2017-05-14_12:11:49.90569 redis6 redis: 54900:M 14 May 12:11:49.905 * MASTER MODE enabled (user request from 'id=10024957 addr=10.45.2.8:39872 fd=349 name= age=394 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=141 qbuf-free=32627 obl=36 oll=0 omem=0 events=r cmd=exec')
2017-05-14_12:11:49.91172 redis6 redis: 54900:M 14 May 12:11:49.911 # CONFIG REWRITE executed with success.
2017-05-14_12:11:51.10397 redis6 redis: 54900:M 14 May 12:11:51.103 * Slave 10.45.2.10:6379 asks for synchronization
2017-05-14_12:11:51.10411 redis6 redis: 54900:M 14 May 12:11:51.104 * Full resync requested by slave 10.45.2.10:6379
2017-05-14_12:11:51.10419 redis6 redis: 54900:M 14 May 12:11:51.104 * Can't attach the slave to the current BGSAVE. Waiting for next BGSAVE for SYNC
2017-05-14_12:12:11.51718 redis6 redis: 54900:M 14 May 12:12:11.517 * Slave 10.45.2.7:6379 asks for synchronization
2017-05-14_12:12:11.51724 redis6 redis: 54900:M 14 May 12:12:11.517 * Full resync requested by slave 10.45.2.7:6379
2017-05-14_12:12:11.51725 redis6 redis: 54900:M 14 May 12:12:11.517 * Can't attach the slave to the current BGSAVE. Waiting for next BGSAVE for SYNC
2017-05-14_12:12:58.04305 redis6 redis: 54900:M 14 May 12:12:58.042 * Slave 10.45.2.6:6379 asks for synchronization
2017-05-14_12:12:58.04315 redis6 redis: 54900:M 14 May 12:12:58.043 * Full resync requested by slave 10.45.2.6:6379
2017-05-14_12:12:58.04322 redis6 redis: 54900:M 14 May 12:12:58.043 * Can't attach the slave to the current BGSAVE. Waiting for next BGSAVE for SYNC
2017-05-14_12:12:59.49459 redis6 redis: 54900:M 14 May 12:12:59.494 * Slave 10.45.2.8:6379 asks for synchronization
2017-05-14_12:12:59.49469 redis6 redis: 54900:M 14 May 12:12:59.494 * Full resync requested by slave 10.45.2.8:6379
2017-05-14_12:12:59.49470 redis6 redis: 54900:M 14 May 12:12:59.494 * Can't attach the slave to the current BGSAVE. Waiting for next BGSAVE for SYNC
2017-05-14_12:17:44.34907 redis6 redis: 56832:C 14 May 12:17:44.348 * DB saved on disk
2017-05-14_12:17:44.84524 redis6 redis: 56832:C 14 May 12:17:44.845 * RDB: 839 MB of memory used by copy-on-write
2017-05-14_12:17:45.66758 redis6 redis: 54900:M 14 May 12:17:45.667 * Background saving terminated with success
2017-05-14_12:17:45.66769 redis6 redis: 54900:M 14 May 12:17:45.667 * Starting BGSAVE for SYNC with target: disk
2017-05-14_12:17:46.23727 redis6 redis: 54900:M 14 May 12:17:46.237 * Background saving started by pid 57780
2017-05-14_12:24:01.61265 redis6 redis: 57780:C 14 May 12:24:01.612 * DB saved on disk
2017-05-14_12:24:02.12719 redis6 redis: 57780:C 14 May 12:24:02.127 * RDB: 371 MB of memory used by copy-on-write
2017-05-14_12:24:02.85458 redis6 redis: 54900:M 14 May 12:24:02.854 * Background saving terminated with success
2017-05-14_12:25:03.02674 redis6 redis: 54900:M 14 May 12:25:03.026 * 10000 changes in 60 seconds. Saving...
2017-05-14_12:25:03.60925 redis6 redis: 54900:M 14 May 12:25:03.609 * Background saving started by pid 59873
2017-05-14_12:31:22.26572 redis6 redis: 54900:M 14 May 12:31:22.265 * Synchronization with slave 10.45.2.7:6379 succeeded
2017-05-14_12:31:22.27279 redis6 redis: 54900:M 14 May 12:31:22.272 * Synchronization with slave 10.45.2.10:6379 succeeded
2017-05-14_12:31:24.83648 redis6 redis: 54900:M 14 May 12:31:24.836 * Synchronization with slave 10.45.2.8:6379 succeeded
2017-05-14_12:31:25.33032 redis6 redis: 54900:M 14 May 12:31:25.330 * Synchronization with slave 10.45.2.6:6379 succeeded
2017-05-14_12:31:36.78192 redis6 redis: 59873:C 14 May 12:31:36.781 * DB saved on disk
2017-05-14_12:31:37.29807 redis6 redis: 59873:C 14 May 12:31:37.297 * RDB: 509 MB of memory used by copy-on-write
```
redis4 logs show:
```
2017-05-14_11:41:33.11078 redis4 redis: 27797:C 14 May 11:41:33.110 * RDB: 621 MB of memory used by copy-on-write
2017-05-14_11:41:34.20420 redis4 redis: 10091:S 14 May 11:41:34.204 * Background saving terminated with success
2017-05-14_11:42:35.08325 redis4 redis: 10091:S 14 May 11:42:35.083 * 10000 changes in 60 seconds. Saving...
2017-05-14_11:42:36.07496 redis4 redis: 10091:S 14 May 11:42:36.074 * Background saving started by pid 28268
2017-05-14_11:43:57.13851 redis4 redis: 10091:M 14 May 11:43:57.138 # Connection with master lost.
2017-05-14_11:43:57.13865 redis4 redis: 10091:M 14 May 11:43:57.138 * Caching the disconnected master state.
2017-05-14_11:43:57.13872 redis4 redis: 10091:M 14 May 11:43:57.138 * Discarding previously cached master state.
2017-05-14_11:43:57.13882 redis4 redis: 10091:M 14 May 11:43:57.138 * MASTER MODE enabled (user request from 'id=121 addr=10.45.2.6:33503 fd=5 name=sentinel-87f8c0f0-cmd age=488017 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=14 qbuf-free=32754 obl=36 oll=0 omem=0 events=r cmd=exec')
2017-05-14_11:43:57.14864 redis4 redis: 10091:M 14 May 11:43:57.148 # CONFIG REWRITE executed with success.
2017-05-14_11:43:58.62633 redis4 redis: 10091:M 14 May 11:43:58.626 * Slave 10.45.2.6:6379 asks for synchronization
2017-05-14_11:43:58.62648 redis4 redis: 10091:M 14 May 11:43:58.626 * Full resync requested by slave 10.45.2.6:6379
2017-05-14_11:43:58.62655 redis4 redis: 10091:M 14 May 11:43:58.626 * Can't attach the slave to the current BGSAVE. Waiting for next BGSAVE for SYNC
2017-05-14_11:44:18.93740 redis4 redis: 10091:M 14 May 11:44:18.937 * Slave 10.45.2.9:6379 asks for synchronization
2017-05-14_11:44:18.93745 redis4 redis: 10091:M 14 May 11:44:18.937 * Full resync requested by slave 10.45.2.9:6379
2017-05-14_11:44:18.93747 redis4 redis: 10091:M 14 May 11:44:18.937 * Can't attach the slave to the current BGSAVE. Waiting for next BGSAVE for SYNC
2017-05-14_11:44:59.07446 redis4 redis: 10091:M 14 May 11:44:59.074 * Slave 10.45.2.10:6379 asks for synchronization
2017-05-14_11:44:59.07456 redis4 redis: 10091:M 14 May 11:44:59.074 * Full resync requested by slave 10.45.2.10:6379
2017-05-14_11:44:59.07461 redis4 redis: 10091:M 14 May 11:44:59.074 * Can't attach the slave to the current BGSAVE. Waiting for next BGSAVE for SYNC
2017-05-14_11:44:59.43544 redis4 redis: 10091:M 14 May 11:44:59.435 * Slave 10.45.2.8:6379 asks for synchronization
2017-05-14_11:44:59.43558 redis4 redis: 10091:M 14 May 11:44:59.435 * Full resync requested by slave 10.45.2.8:6379
2017-05-14_11:44:59.43565 redis4 redis: 10091:M 14 May 11:44:59.435 * Can't attach the slave to the current BGSAVE. Waiting for next BGSAVE for SYNC
2017-05-14_11:48:00.54069 redis4 redis: 28268:C 14 May 11:48:00.540 * DB saved on disk
2017-05-14_11:48:01.33974 redis4 redis: 28268:C 14 May 11:48:01.339 * RDB: 795 MB of memory used by copy-on-write
2017-05-14_11:48:02.46376 redis4 redis: 10091:M 14 May 11:48:02.463 * Background saving terminated with success
2017-05-14_11:48:02.46387 redis4 redis: 10091:M 14 May 11:48:02.463 * Starting BGSAVE for SYNC with target: disk
2017-05-14_11:48:03.44009 redis4 redis: 10091:M 14 May 11:48:03.440 * Background saving started by pid 28707
2017-05-14_11:53:37.81787 redis4 redis: 28707:C 14 May 11:53:37.816 * DB saved on disk
2017-05-14_11:53:38.61419 redis4 redis: 28707:C 14 May 11:53:38.614 * RDB: 1253 MB of memory used by copy-on-write
2017-05-14_11:53:39.76269 redis4 redis: 10091:M 14 May 11:53:39.762 * Background saving terminated with success
2017-05-14_11:54:40.02433 redis4 redis: 10091:M 14 May 11:54:40.024 * 10000 changes in 60 seconds. Saving...
2017-05-14_11:54:40.99628 redis4 redis: 10091:M 14 May 11:54:40.996 * Background saving started by pid 29443
2017-05-14_12:01:02.60096 redis4 redis: 10091:M 14 May 12:01:02.600 # Background saving terminated by signal 9
2017-05-14_12:01:03.14139 redis4 redis: 10091:M 14 May 12:01:03.141 * 10 changes in 300 seconds. Saving...
2017-05-14_12:01:04.13781 redis4 redis: 10091:M 14 May 12:01:04.137 * Background saving started by pid 31628
2017-05-14_12:01:21.10639 redis4 redis: 10091:M 14 May 12:01:21.106 * Synchronization with slave 10.45.2.10:6379 succeeded
2017-05-14_12:01:21.11380 redis4 redis: 10091:M 14 May 12:01:21.113 * Synchronization with slave 10.45.2.6:6379 succeeded
2017-05-14_12:01:21.12017 redis4 redis: 10091:M 14 May 12:01:21.120 * Synchronization with slave 10.45.2.8:6379 succeeded
2017-05-14_12:01:21.12330 redis4 redis: 10091:M 14 May 12:01:21.123 * Synchronization with slave 10.45.2.9:6379 succeeded
2017-05-14_12:06:54.21156 redis4 redis: 31628:C 14 May 12:06:54.211 * DB saved on disk
2017-05-14_12:06:55.02548 redis4 redis: 31628:C 14 May 12:06:55.025 * RDB: 587 MB of memory used by copy-on-write
2017-05-14_12:06:56.62981 redis4 redis: 10091:M 14 May 12:06:56.629 * Background saving terminated with success
2017-05-14_12:07:57.08709 redis4 redis: 10091:M 14 May 12:07:57.087 * 10000 changes in 60 seconds. Saving...
2017-05-14_12:07:58.04379 redis4 redis: 10091:M 14 May 12:07:58.043 * Background saving started by pid 32168
2017-05-14_12:11:39.17502 redis4 redis: 10091:M 14 May 12:11:39.173 # Background saving terminated by signal 9
2017-05-14_12:11:39.55882 redis4 redis: 10091:M 14 May 12:11:39.558 * 10000 changes in 60 seconds. Saving...
2017-05-14_12:11:40.54053 redis4 redis: 10091:M 14 May 12:11:40.540 * Background saving started by pid 32649
2017-05-14_12:12:10.89896 redis4 redis: 10091:S 14 May 12:12:10.898 # Connection with slave 10.45.2.6:6379 lost.
2017-05-14_12:12:10.98561 redis4 redis: 10091:S 14 May 12:12:10.985 # Connection with slave 10.45.2.9:6379 lost.
2017-05-14_12:12:10.98576 redis4 redis: 10091:S 14 May 12:12:10.985 # Connection with slave 10.45.2.10:6379 lost.
2017-05-14_12:12:11.05003 redis4 redis: 10091:S 14 May 12:12:11.049 # Connection with slave 10.45.2.8:6379 lost.
2017-05-14_12:12:11.05038 redis4 redis: 10091:S 14 May 12:12:11.050 * SLAVE OF 10.45.2.9:6379 enabled (user request from 'id=211999 addr=10.45.2.10:48072 fd=10 name=sentinel-32fcc701-cmd age=20 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=0 qbuf-free=32768 obl=36 oll=0 omem=0 events=r cmd=exec')
2017-05-14_12:12:11.06520 redis4 redis: 10091:S 14 May 12:12:11.065 # CONFIG REWRITE executed with success.
2017-05-14_12:12:11.51464 redis4 redis: 10091:S 14 May 12:12:11.514 * Connecting to MASTER 10.45.2.9:6379
2017-05-14_12:12:11.51472 redis4 redis: 10091:S 14 May 12:12:11.514 * MASTER <-> SLAVE sync started
2017-05-14_12:12:11.51524 redis4 redis: 10091:S 14 May 12:12:11.515 * Non blocking connect for SYNC fired the event.
2017-05-14_12:12:11.51555 redis4 redis: 10091:S 14 May 12:12:11.515 * Master replied to PING, replication can continue...
2017-05-14_12:12:11.51621 redis4 redis: 10091:S 14 May 12:12:11.516 * Partial resynchronization not possible (no cached master)
2017-05-14_12:16:43.74981 redis4 redis: 32649:C 14 May 12:16:43.749 * DB saved on disk
2017-05-14_12:16:44.49529 redis4 redis: 32649:C 14 May 12:16:44.495 * RDB: 524 MB of memory used by copy-on-write
2017-05-14_12:16:45.52604 redis4 redis: 10091:S 14 May 12:16:45.525 * Background saving terminated with success
2017-05-14_12:17:46.23717 redis4 redis: 10091:S 14 May 12:17:46.237 * Full resync from master: 405b30be07f1a319117af61bbb7ceb46665f0155:127939518653
2017-05-14_12:24:02.85872 redis4 redis: 10091:S 14 May 12:24:02.858 * MASTER <-> SLAVE sync: receiving 17638136249 bytes from master
2017-05-14_12:31:22.69705 redis4 redis: 10091:S 14 May 12:31:22.696 * MASTER <-> SLAVE sync: Flushing old data
2017-05-14_12:35:14.05229 redis4 redis: 10091:S 14 May 12:35:14.052 * MASTER <-> SLAVE sync: Loading DB in memory
2017-05-14_12:41:31.10755 redis4 redis: 10091:S 14 May 12:41:31.107 * MASTER <-> SLAVE sync: Finished with success
2017-05-14_12:41:31.10763 redis4 redis: 10091:S 14 May 12:41:31.107 * 1 changes in 900 seconds. Saving...
2017-05-14_12:41:31.67129 redis4 redis: 10091:S 14 May 12:41:31.671 * Background saving started by pid 36948
2017-05-14_12:47:09.42127 redis4 redis: 36948:C 14 May 12:47:09.421 * DB saved on disk
2017-05-14_12:47:09.89025 redis4 redis: 36948:C 14 May 12:47:09.890 * RDB: 897 MB of memory used by copy-on-write
2017-05-14_12:47:10.61618 redis4 redis: 10091:S 14 May 12:47:10.616 * Background saving terminated with success
2017-05-14_12:48:11.07917 redis4 redis: 10091:S 14 May 12:48:11.079 * 10000 changes in 60 seconds. Saving...
2017-05-14_12:48:11.67863 redis4 redis: 10091:S 14 May 12:48:11.678 * Background saving started by pid 37485
2017-05-14_12:53:47.26943 redis4 redis: 37485:C 14 May 12:53:47.269 * DB saved on disk
2017-05-14_12:53:47.73025 redis4 redis: 37485:C 14 May 12:53:47.730 * RDB: 318 MB of memory used by copy-on-write
2017-05-14_12:53:48.36025 redis4 redis: 10091:S 14 May 12:53:48.360 * Background saving terminated with success
2017-05-14_12:54:49.03322 redis4 redis: 10091:S 14 May 12:54:49.033 * 10000 changes in 60 seconds. Saving...
2017-05-14_12:54:49.62130 redis4 redis: 10091:S 14 May 12:54:49.621 * Background saving started by pid 38006
```
issue