postgres-01 and postgres-04 are not in the repmgr cluster
It seems that postgres-01 and postgres-04 are not part of the repmgr cluster:
$ sudo gitlab-ctl repmgr cluster show
Role | Name | Upstream | Connection String
----------+-------------------------------|-------------------------------|-------------------------------------------------------------------------------------
* master | postgres-02.db.prd.gitlab.com | | host=postgres-02.db.prd.gitlab.com port=5432 user=gitlab_repmgr dbname=gitlab_repmgr
standby | postgres-03.db.prd.gitlab.com | postgres-02.db.prd.gitlab.com | host=postgres-03.db.prd.gitlab.com port=5432 user=gitlab_repmgr dbname=gitlab_repmgr
I tried adding postgres-04 using sudo gitlab-ctl repmgr standby register but this spits out:
Error running command: /opt/gitlab/embedded/bin/repmgr -f /var/opt/gitlab/postgresql/repmgr.conf standby register
ERROR: INFO: connecting to standby database
ERROR: this node should be a standby (host=postgres-02.db.prd.gitlab.com port=5432 user=gitlab_repmgr dbname=gitlab_repmgr)
These two hosts should be added back to the cluster.