Sidekiq RPS anomalously low
Summary
We received alerts for sidekiq's request rate being anomalously low. These metrics are derived from logs. Searching our elasticsearch logging cluster revealed that the problem was likely in our metrics themselves, rather than the request rate actually being so low.
Noticed by @andrewn.
Timeline
2019-07-22
- 06:06 UTC - The sidekiq requests per second drops sharply and quickly becomes anomalously low.
- 09:30 UTC - The on-call engineer begins looking into the issue.
- 11:12 UTC - The issue is resolved.
Edited by Craig Furman