Geo: Secondary stop syncing until restart

Summary

Geo secondary will stop syncing and replication slot will go inactive. After restarting the secondary, replication will take place and the replication slot will show active for a while before stopping again.

Relevant logs and/or screenshots

Postgres logs on the primary:

2018-03-22_19:51:08.77836 LOG: could not receive data from client: Connection reset by peer

2018-03-22_19:51:08.77840 LOG: unexpected EOF on standby connection

Postgres logs on the secondary:

2018-03-22_19:51:08.65907 LOG: incorrect resource manager data checksum in record at 62B/D0FAD9C0

2018-03-22_19:51:08.66788 FATAL: terminating walreceiver process due to administrator command

2018-03-22_19:51:08.67413 LOG: incorrect resource manager data checksum in record at 62B/D0FAD9C0

2018-03-22_19:51:08.67417 LOG: incorrect resource manager data checksum in record at 62B/D0FAD9C0

Log location (62B/D0FAD9C0) changes each time this happens after a restart.

GitLab 10.5.3-ee

Customer ticket -> https://gitlab.zendesk.com/agent/tickets/93135 (internal) Follow-up ticket -> https://gitlab.zendesk.com/agent/tickets/97312 (internal)