2020-04-30 Workhorse canary component apdex-score burn rate outside of SLO
Summary
An automated job of some kind, from a single IP, hit the same URL several hundred times in less than 2 minutes, and received a 'Deadline Exceeded'. Combined with lower activity on canary, this caused the overall error rate to dip too low, and paged. There was no systemic issue
Timeline
All times UTC.
2020-04-30
- 06:45 - Page for `The workhorse component of the web service, (cny stage), has a apdex-score burn rate outside of SLO"
- 06:55 - From logs, identified the cause as transient, not systemic, and ceased.
Waiting for the alert to clear, as the dip was severe enough numerically to affect the SLO for a long time.