2020-08-26: Deployments blocked by sentry errors
Summary
Deployments blocked by sentry errors
The latest auto-deploy branches are seeing lots of new errors
Timeline
All times UTC.
2020-08-26
- 09:24 - aphillips declares incident in Slack using
/incident declare
command. - 09:40 - Imre Farkas (as on call dev escalation) creates a revert MR - gitlab-org/gitlab!40494 (merged)
- 21:00 - MR makes it through to staging and mitigates the issue
Click to expand or collapse the Incident Review section.
Incident Review
New Sentry errors were detected on canary. Examples:
- https://sentry.gitlab.net/gitlab/gitlabcom/issues/1783035/ and
- https://sentry.gitlab.net/gitlab/gitlabcom/issues/1783057/
After discussing with the dev on call engineer (Imre Farkas) the decision was made to revert the commit causing the errors to unblock deployments.
Summary
Lessons Learned
We should always revert if we're unsure of the impact errors might have.
Corrective Actions
N/A
Guidelines
Edited by Amy Phillips