Severe site degradation due to database load
At around 01:26 UTC, a database failover was accidentally performed leading to a split-brain problem. Fortunately, the fleet continued to follow the true primary.
We shut down postgres-01 since it was the rogue primary. In our investigation, both postgres-03 and postgres-04 were trying to follow postgres-01. As such, we are rebuilding replication on postgres-03 as I write this issue and then postgres-04 when it is finished.
Unfortunately, progress is slowed since we are also simultaneously taking a pg_basebackup for WAL-E since we do not have a full basebackup since the correct failover earlier today.
This also means that ALL read traffic is going to postgres-02, which has led to some slow performance, however the site remains up. We have also stopped sidekiq-cluster on the gprd besteffort nodes as they cause a very large query to be performed on the database that can't be handled at this time.
cc/ @gl-infra