2026-02-04: The loadbalancer SLI of the ci-runners service (main stage) has an error rate violating SLO

The loadbalancer SLI of the ci-runners service (main stage) has an error rate violating SLO (Severity 3)

Problem: The load balancer for the ci-runners service was returning an elevated number of errors, violating the service level objective.

Impact: The ci-runners service experienced a sustained error rate above the service level objective, peaking at 3.1%. Users may have seen failed or delayed job updates during this period.

Causes: A recent merge request related to CI job failure reasons introduced a bug. The runner API 'update_job' endpoint encountered an undefined method 'to_sym' on a build status reason object, causing HTTP 5xx errors.

Response strategy: We merged a revert to undo the faulty change and deployed the fix. After deployment, error rates dropped to normal levels.

This ticket was created to track INC-7116, by incident.io 🔥