2020-05-12 Gitaly canary slowdown
Summary
Gitaly canary slowdown. Activity on gitlab-org/gitlab was very and caused performance issues on interactions with that project. It appears this was due to an initial issue that then revealed further issues linked to this production issue.
Timeline
All times UTC.
2020-05-13
- 13:51 - Incident declared from Slack
- 17:20 - Most Gitaly Canary node interactions back to normal.
Details
Mostly recovered. Most of discussion done in Incident zoom and #incident-management.
Source
Incident declared by t4cc0re in Slack via /incident declare
command.
Resources
- If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)
Edited by Dave Smith