2026-04-04: Web service error rate violating SLO on loadbalancer, rails_request, and workhorse components

GitLab Infrastructure Service - incident.io
2026-04-04T11:54:28Z

# Web service error rate violating SLO on loadbalancer, rails_request, and workhorse components (Severity 3)

**Problem**: The web service is experiencing HTTP 5xx error rates above the defined SLO for the past 6 hours, with a 0.4326% error rate observed on the load balancer and similar issues in rails_request and workhorse components.

**Impact**: Thousands of requests have caused the web service's error rate to exceed SLO thresholds, primarily affecting access to major repositories. Almost half the traffic is from the US and over a third from Vietnam.

**Causes**: A high volume of anonymous requests from a wide variety of IPs and user agents are targeting popular repositories, resulting in increased load and elevated HTTP 5xx error rates.

**Response strategy**: We investigated options for rate limiting by project path but current tools do not support project-based grouping. No application-level changes have been made due to tool limitations and lack of safe options for immediate rollout.

_This ticket was created to track_ [_INC-8997_](https://app.incident.io/gitlab/incidents/8997)_, by_ [_incident.io_](https://app.incident.io) 🔥