2025-10-06: Requests queuing on internal-api in us-east1 due to no free Puma workers

Requests queuing on internal-api in us-east1 due to no free Puma workers (Severity 4 (Low))

Problem: Requests to the internal-api service in us-east1 were delayed due to no free Puma workers, causing requests to queue.

Impact: Requests to the internal-api service in us-east1 experienced queuing delays, with the rails_queueing apdex dropping to 95.82% over 6 hours, violating the SLO.

Causes: An HAProxy unattended upgrade unintentionally re-enabled the canary deployment, which led to stale connections and requests being queued for free Puma workers. This was only two stuck connections on a single HAProxy Node

Response strategy: We re-drained the canary deployment to clear unexpected traffic and are waiting to re-enable canary after related load balancer issues are resolved.

---

This ticket was created to track INC-4494, by incident.io 🔥