2020-08-31: Push mirrors failing
Summary
Push mirrors failing
Push Mirrors of repos on GitLab.com have been failing/running into issues related to gitlab-org/gitlab#242061 (closed) - we should post to https://status.gitlab.com and link issue
Timeline
All times UTC.
2020-08-31
- 15:13 - dsmith declares incident in Slack using
/incident declare
command. - 17:00 - dsmith -tracking/discussion happening on gitlab-org/gitlab#242061 (closed)
- 19:17 - bumped to S2 based on definitions and to make sure an incident review is created
- 19:59 - patch is in process of rolling out. Initial spot checks are that it is working. Will post to status when it is finished rolling out.
- 20:22 - initial confirmation that the patch has things working again. Moving to mitigated.
Click to expand or collapse the Incident Review section.
Incident Review
Summary
- Service(s) affected:
- Team attribution:
- Minutes downtime or degradation:
Metrics
Customer Impact
- Who was impacted by this incident? (i.e. external customers, internal customers)
- What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)
- How many customers were affected?
- If a precise customer impact number is unknown, what is the estimated potential impact?
Incident Response Analysis
- How was the event detected?
- How could detection time be improved?
- How did we reach the point where we knew how to mitigate the impact?
- How could time to mitigation be improved?
Post Incident Analysis
- How was the root cause diagnosed?
- How could time to diagnosis be improved?
- Do we have an existing backlog item that would've prevented or greatly reduced the impact of this incident?
- Was this incident triggered by a change (deployment of code or change to infrastructure. If yes, have you linked the issue which represents the change?)?
5 Whys
Lessons Learned
Corrective Actions
Guidelines
Edited by Brent Newton