2026-02-05: Loadbalancer error rate for web service (cny stage) violating SLO

Loadbalancer error rate for web service (cny stage) violating SLO (Severity 4)

Problem: The load balancer for the web service in the cny stage saw elevated HTTP 5xx errors, breaching its error rate SLO.

Impact: Users of the web service in the cny stage experienced a temporary increase in failed requests, with HTTP 5xx error rates peaking at around 0.17%, exceeding the service level objective.

Causes: Most errors were isolated to a single Kubernetes pod, which showed 'badgateway' errors related to EOF while waiting for a response. The problematic pod was removed, likely as part of a routine deployment. The specific upstream cause remains unclear.

Response strategy: A new deployment to the cny stage completed, which removed the problematic pod. The error rate has since dropped and all related alerts are resolved.

This ticket was created to track INC-7155, by incident.io 🔥