At around 01:26 UTC, a database failover was accidentally performed leading to a split-brain problem. Fortunately, the fleet continued to follow the true primary. We shut down `postgres-01` since it was the rogue primary. In our investigation, both `postgres-03` and `postgres-04` were trying to follow `postgres-01`. As such, we are rebuilding replication on `postgres-03` as I write this issue and then `postgres-04` when it is finished. Unfortunately, progress is slowed since we are also simultaneously taking a pg_basebackup for WAL-E since we do not have a full basebackup since the correct failover earlier today. This also means that ALL read traffic is going to `postgres-02`, which has led to some slow performance, however the site remains up. We have also stopped `sidekiq-cluster` on the gprd besteffort nodes as they cause a very large query to be performed on the database that can't be handled at this time. cc/ @gl\-infra