2026-05-19: Redis primary CPU saturation on redis-sidekiq nodes approaching capacity limit

Redis primary CPU saturation on redis-sidekiq nodes approaching capacity limit (Severity 2)

Problem: Primary Redis node for redis-sidekiq shard catchall_b exceeded CPU and memory limits, creating a risk of service degradation and potential outage.

Impact: For about 3 hours (from 15:48 to 18:54 UTC), audit event jobs were dropped globally. This resulted in missing audit event records for all customers using audit events, and customers may have experienced delays or drops in background job processing during this window.

Causes: A surge in audit event jobs, with many lacking namespace information, overwhelmed the redis-sidekiq catchall_b shard. This led to both CPU and memory saturation on the primary Redis node. Redis's single-threaded design limited scaling to a single CPU core, contributing to the resource bottleneck.

Response strategy: We switched from deferring to dropping audit event jobs globally, which quickly reduced load and resolved CPU and memory saturation. Audit event processing was resumed after approximately 3 hours, restoring normal operations.

This ticket was created to track INC-10255, by incident.io 🔥