2026-05-06: Loadbalancer error rate for api service in cny stage exceeds SLO (3.7%)

Loadbalancer error rate for api service in cny stage exceeds SLO (3.7%) (Severity 2)

Problem: The load balancer for the api service in the cny stage is returning 3.7% HTTP 5xx errors over the past hour, exceeding the service level objective.

Impact: Users on GitLab Next (canary) experienced errors, such as being unable to load dropdown lists or perform some actions. Regular production users are not affected.

Causes: A configuration update that routed canary traffic away from NGINX caused increased 5xx errors on the HAProxy load balancer for the api service in the cny stage.

Response strategy: We reverted the configuration change that routed canary traffic away from NGINX. The team forced a canary drain and ran Chef client across all HAProxy nodes to restore a known-good state. State persistence was re-enabled, and error rates are being monitored as canary traffic returns.

This ticket was created to track INC-9883, by incident.io 🔥