Severe site degradation due to database load

At around 01:26 UTC, a database failover was accidentally performed leading to a split-brain problem. Fortunately, the fleet continued to follow the true primary.

We shut down postgres-01 since it was the rogue primary. In our investigation, both postgres-03 and postgres-04 were trying to follow postgres-01. As such, we are rebuilding replication on postgres-03 as I write this issue and then postgres-04 when it is finished.

Unfortunately, progress is slowed since we are also simultaneously taking a pg_basebackup for WAL-E since we do not have a full basebackup since the correct failover earlier today.

This also means that ALL read traffic is going to postgres-02, which has led to some slow performance, however the site remains up. We have also stopped sidekiq-cluster on the gprd besteffort nodes as they cause a very large query to be performed on the database that can't be handled at this time.

cc/ @gl-infra