2026-02-04: The polling SLI of the ci-runners service (main stage) has an error rate violating SLO

The polling SLI of the ci-runners service (main stage) has an error rate violating SLO (Severity 4)

Problem: The polling SLI for the ci-runners service experienced a short-lived spike in errors, briefly violating the service level objective (SLO).

Impact: A spike in errors for the ci-runners polling service caused the error rate to briefly reach nearly 75% between 08:06:50 and 08:07:50. Traffic levels were very low during this time.

Causes: A restart of the single-node Redis master pod for the ci-runners service (with no high availability) occurred just before the error window. During this time, runner polling requests failed due to Redis connectivity errors while reading stickiness keys during authentication.

Response strategy: Metrics and service behavior returned to normal immediately after the Redis master restart.

This ticket was created to track INC-7084, by incident.io 🔥