Severe site degradation due to database load
At around 01:26 UTC, a database failover was accidentally performed leading to a split-brain problem. Fortunately, the fleet continued to follow the true primary.
We shut down
postgres-01 since it was the rogue primary. In our investigation, both
postgres-04 were trying to follow
postgres-01. As such, we are rebuilding replication on
postgres-03 as I write this issue and then
postgres-04 when it is finished.
Unfortunately, progress is slowed since we are also simultaneously taking a pg_basebackup for WAL-E since we do not have a full basebackup since the correct failover earlier today.
This also means that ALL read traffic is going to
postgres-02, which has led to some slow performance, however the site remains up. We have also stopped
sidekiq-cluster on the gprd besteffort nodes as they cause a very large query to be performed on the database that can't be handled at this time.