2025-10-10: Loadbalancer 5xx error rate violating SLO for web service main stage

Loadbalancer 5xx error rate violating SLO for web service main stage (Severity 4 (Low))

Problem: The load balancer for the web service main stage reported a sudden spike in 5xx errors, briefly exceeding the SLO for HTTP request failures.

Impact: There was a short-lived 0.2179% error rate for HTTP requests in the web service's main stage load balancer. The actual number of affected users is unclear, but the incident happened during off-peak hours and did not appear to impact downstream services.

Causes: A brief, significant spike in HAProxy connections and outgoing traffic occurred, causing the error rate to exceed the SLO. This spike was isolated and not accompanied by increased request rates or issues in downstream components.

Response strategy: We investigated the spike, reviewed HAProxy and network traffic data, and confirmed that the error rate returned to normal.

---

This ticket was created to track INC-4697, by incident.io 🔥