2020-03-15 Database Role Change
Summary
The production database appears to have failed over to patroni-01 at 2020-03-15 04:10:00 UTC. This created a large spike in errors, but returned to normal fairly quickly
More information will be added as we investigate the issue.
Timeline
All times UTC.
2020-03-15 UTC
- 40:10:00 - Alert for increased backend error rates - https://gitlab.pagerduty.com/incidents/PVBVDO6
- 04:10:00 -
patroni-06
alerted that it was down - https://gitlab.pagerduty.com/incidents/PP43YEL - 04:10:00 -
patroni-01
took over as master
Resources
- If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)
Edited by Devin Sylva