Sign in or sign up before continuing. Don't have an account yet? Register now to get started.
Register now

redis-cache cluster broken

currently redis-cache-03 is the master,

redis-cache-02 and 01 can not sync against it:

looping:

2018-03-19_12:54:43.02692 redis-cache-02 redis: 70418:S 19 Mar 12:54:43.026 * Connecting to MASTER 10.66.2.13:6379
2018-03-19_12:54:43.02719 redis-cache-02 redis: 70418:S 19 Mar 12:54:43.027 * MASTER <-> SLAVE sync started
2018-03-19_12:54:43.02781 redis-cache-02 redis: 70418:S 19 Mar 12:54:43.027 * Non blocking connect for SYNC fired the event.
2018-03-19_12:54:43.13928 redis-cache-02 redis: 70418:S 19 Mar 12:54:43.139 * Master replied to PING, replication can continue...
2018-03-19_12:54:43.16786 redis-cache-02 redis: 70418:S 19 Mar 12:54:43.167 * Partial resynchronization not possible (no cached master)
2018-03-19_12:54:45.76288 redis-cache-02 redis: 70418:S 19 Mar 12:54:45.762 * Full resync from master: d5633d153a852c34a760ac7011f97b8de978226b:256695183257
2018-03-19_13:04:31.88655 redis-cache-02 redis: 70418:S 19 Mar 13:04:31.886 # I/O error reading bulk count from MASTER: Resource temporarily unavailable
2018-03-19_13:04:32.23269 redis-cache-02 redis: 70418:S 19 Mar 13:04:32.232 * Connecting to MASTER 10.66.2.13:6379
2018-03-19_13:04:32.23290 redis-cache-02 redis: 70418:S 19 Mar 13:04:32.232 * MASTER <-> SLAVE sync started
2018-03-19_13:04:32.23359 redis-cache-02 redis: 70418:S 19 Mar 13:04:32.233 * Non blocking connect for SYNC fired the event.
2018-03-19_13:04:34.55052 redis-cache-02 redis: 70418:S 19 Mar 13:04:34.550 * Master replied to PING, replication can continue...
2018-03-19_13:04:34.59785 redis-cache-02 redis: 70418:S 19 Mar 13:04:34.597 * Partial resynchronization not possible (no cached master)
2018-03-19_13:04:34.60278 redis-cache-02 redis: 70418:S 19 Mar 13:04:34.602 * Full resync from master: d5633d153a852c34a760ac7011f97b8de978226b:257103162953

with:

2018-03-19_12:54:43.16899 redis-cache-03 redis: 32372:M 19 Mar 12:54:43.168 * Slave 10.66.2.12:6379 asks for synchronization                                                                                        
2018-03-19_12:54:43.16906 redis-cache-03 redis: 32372:M 19 Mar 12:54:43.168 * Full resync requested by slave 10.66.2.12:6379                                                                                        
2018-03-19_12:54:43.16913 redis-cache-03 redis: 32372:M 19 Mar 12:54:43.168 * Starting BGSAVE for SYNC with target: disk                                                                                            
2018-03-19_12:54:45.76207 redis-cache-03 redis: 32372:M 19 Mar 12:54:45.761 * Background saving started by pid 53906                                                                                                
2018-03-19_12:54:45.89939 redis-cache-03 redis: 32372:M 19 Mar 12:54:45.899 * Slave 10.66.2.11:6379 asks for synchronization                                                                                        
2018-03-19_12:54:45.89962 redis-cache-03 redis: 32372:M 19 Mar 12:54:45.899 * Full resync requested by slave 10.66.2.11:6379                                                                                        
2018-03-19_12:54:45.89977 redis-cache-03 redis: 32372:M 19 Mar 12:54:45.899 * Waiting for end of BGSAVE for SYNC                                                                                                    
2018-03-19_13:03:38.71114 redis-cache-03 redis: 32372:M 19 Mar 13:03:38.711 # Client id=3229257 addr=10.66.2.12:37543 fd=2199 name= age=536 idle=536 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=16232 oll=19217 omem=536882808 events=r cmd=psync scheduled to be closed ASAP for overcoming of output buffer limits.                                                                                                    
2018-03-19_13:03:38.71142 redis-cache-03 redis: 32372:M 19 Mar 13:03:38.711 # Client id=3229284 addr=10.66.2.11:46465 fd=2241 name= age=533 idle=533 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=16232 oll=19217 omem=536886904 events=r cmd=psync scheduled to be closed ASAP for overcoming of output buffer limits.                                                                                                    
2018-03-19_13:03:38.80154 redis-cache-03 redis: 32372:M 19 Mar 13:03:38.801 # Connection with slave 10.66.2.12:6379 lost.                                                                                           
2018-03-19_13:03:39.08773 redis-cache-03 redis: 32372:M 19 Mar 13:03:39.087 # Connection with slave 10.66.2.11:6379 lost.

in the master's logfile

Assignee Loading
Time tracking Loading