2026-02-06: Workhorse error rate violating SLO in us-east1-b region

Workhorse error rate violating SLO in us-east1-b region (Severity 3)

Problem: A spike in error rate for web requests through the workhorse component in us-east1-b was detected, breaching the service level objective.

Impact: For the past hour, 0.146% of web requests in the us-east1-b region failed, exceeding the error rate SLO. The issue was isolated to requests routed through the affected pod.

Causes: Nearly all 5xx errors were coming from a single webservice pod in us-east1-b, which was returning both 500 and 200 responses. The errors were dominated by ArgumentError exceptions. Other pods were unaffected.

Response strategy: The problematic pod was removed by a new deployment, which resolved the elevated error rates and cleared all related alerts.

This ticket was created to track INC-7175, by incident.io 🔥