2026-06-05: The loadbalancer SLI of the ci-runners service (main stage) has an error rate violating SLO

The loadbalancer SLI of the ci-runners service (main stage) has an error rate violating SLO (Severity 1)

Problem: A production deployment of both a HAProxy version bump and a traffic routing change caused a spike in 5xx errors on the ci-runners load balancer, stopping all CI job processing for Linux runners.

Impact: From 11:22am UTC, the majority of CI jobs for customer-facing Linux runners and GitLab's internal runners stopped processing, preventing users from running pipelines. The job backlog peaked at over 30,000 jobs. MacOS, ARM64, and GPU runners were not affected. As of 13:30pm UTC, error rates have returned to normal and fewer than 1,000 jobs remain in the queue, which is now being processed at normal speed.

Causes: A HAProxy version bump and a traffic routing change were deployed in close succession to production. These changes introduced breaking issues to the ci-runners load balancer, causing increased 5xx errors and halting CI job processing. Both changes were reverted, which resolved the issue.

Response strategy: We reverted both the HAProxy version and the routing changes across all CI load balancer nodes using a rollback and manual Chef client runs. After these reverts, error rates returned to normal and CI jobs resumed processing. The job backlog has dropped significantly, with recovery confirmed as of 13:30 UTC.

This ticket was created to track INC-10792, by incident.io 🔥