2026-05-22: Workhorse web requests in us-east1-b exceeding error rate SLO
# Workhorse web requests in us-east1-b exceeding error rate SLO (Severity 2) **Problem**: Automated bot traffic triggered persistent HTTP 500 errors and routing exceptions on merge request discussions JSON endpoints containing legacy data, causing web error rate SLO violations and blocking deployments. **Impact**: Error rates for the web service spiked, with over 99% of recent 5xx errors coming from a few old public merge requests. This caused SLO violations, blocked deployments for all customers, and delayed patch and security releases. After blocking the offending bot traffic, the error rate dropped sharply below SLO thresholds and deployments have resumed. **Causes**: Automated bot traffic (notably with the 'Go-http-client/1.1' user agent) repeatedly hit old public merge request discussion endpoints containing legacy or malformed data, surfacing a series of edge-case failures. These included various HTTP 500 errors triggered by unexpected data formats, missing commits, and invalid UTF-8 in URLs. Each fix revealed new legacy data issues, requiring further defensive code changes and targeted traffic blocking. **Response strategy**: - Multiple defensive fixes were merged and deployed to handle different legacy data errors (including missing commits and invalid UTF-8). • Cloudflare rules were applied to block bot traffic hitting the most affected endpoints, resulting in a sharp drop in error rates and restored service health. _This ticket was created to track_ [_INC-10379_](https://app.incident.io/gitlab/incidents/10379)_, by_ [_incident.io_](https://app.incident.io) 🔥
issue