2025-12-01: Sidekiq queueing SLO violation on catchall-b shard (apdex at 60.44%)

Sidekiq queueing SLO violation on catchall-b shard (apdex at 60.44%) (Severity 3)

Problem: A spike in job volume caused only 60.44% of Sidekiq jobs on the catchall-b shard to meet their queueing duration targets, resulting in an SLO violation.

Impact: Sidekiq jobs on the catchall-b shard experienced degraded performance, with apdex dropping to 60.44% and the queue length peaking close to 700,000 jobs. This resulted in small delays for job processing during the incident window.

Causes: The 'Audit Stream IP restrictions' feature, introduced in a recent deployment, caused the spike in Sidekiq job volume and queueing delays.

Response strategy: We reverted the 'Audit Stream IP restrictions' feature and began deploying the fix across all environments.

---

This ticket was created to track INC-6058, by incident.io 🔥