2025-12-02: Increased request queuing in API due to no free Puma workers
Increased request queuing in API due to no free Puma workers (Severity 3)
Problem: Requests to the API service in the cny stage have been queuing because there were no free Puma workers available.
Impact: Users experienced higher latency when accessing the API in the cny stage. The Apdex for rails queueing dropped to as low as 96.98%, triggering an SLO violation. Despite mitigation efforts, periods of degraded performance continue, especially in the cny environment due to its smaller scale.
Causes: High-volume, deeply paginated API requests from a small set of users led to all Puma workers for the API service in the cny stage being busy. The Horizontal Pod Autoscaler was maxed out, but overall traffic did not increase. Three IPs were identified as consistently sending requests with page numbers over 10,000, accounting for millions of events.
Response strategy: We implemented a Cloudflare logging rule and then rolled out a Cloudflare rate limiting rule targeting high-volume paginated API requests. While these mitigations identified the correct sources of traffic, they have not fully resolved the queuing and performance issues. We are continuing to monitor the situation and consider further mitigation options.
This ticket was created to track INC-6074, by incident.io