Patroni node can not replicate data after failover due to timeline before recovery point
A prospect is trying to setup Postgre HA by following the document at https://docs.gitlab.com/ee/administration/postgresql/replication_and_failover.html#patroni. After initial setup the replication from leader to secondary worked well.
Then they attempted to reboot the leader to verify the automated failover. One of the previous secondary nodes was successfully promoted to leader and the previous leader became a secondary. But this previous leader can not replicate data from new leader.
And the patroni log on this node shows as follows.
2020-10-12_09:51:11.41870 LOG: replication terminated by primary server
2020-10-12_09:51:11.41872 DETAIL: End of WAL reached on timeline 1 at 38/C856EFA0.
2020-10-12_09:51:11.41896 LOG: new timeline 2 forked off current database system timeline 1 before current recovery point 38/C856F010
2020-10-12_09:51:14.42107 2020-10-12 17:51:14,420 INFO: Lock owner: my-git-cluster-07; I am my-git-cluster-09
2020-10-12_09:51:14.42112 2020-10-12 17:51:14,420 INFO: does not have lock
2020-10-12_09:51:14.43525 2020-10-12 17:51:14,434 INFO: no action. i am a secondary and i am following a leader
2020-10-12_09:51:16.41276 LOG: restarted WAL streaming at 38/C8000000 on timeline 1
2020-10-12_09:51:16.42224 LOG: replication terminated by primary server
2020-10-12_09:51:16.42225 DETAIL: End of WAL reached on timeline 1 at 38/C856EFA0.
2020-10-12_09:51:16.42246 LOG: new timeline 2 forked off current database system timeline 1 before current recovery point 38/C856F010
2020-10-12_09:51:21.41669 LOG: restarted WAL streaming at 38/C8000000 on timeline 1
2020-10-12_09:51:21.42646 LOG: replication terminated by primary server
2020-10-12_09:51:21.42647 DETAIL: End of WAL reached on timeline 1 at 38/C856EFA0.
2020-10-12_09:51:21.42681 LOG: new timeline 2 forked off current database system timeline 1 before current recovery point 38/C856F010
Why this happens and any way to prevent it?
Edited by Xiaogang Wen