2025-09-22: High error rate for job polling on /api/v4/jobs/request in ci-runners main stage

High error rate for job polling on /api/v4/jobs/request in ci-runners main stage (Severity 4 (Low))

Problem: High error rate (18.88%) for job polling requests to /api/v4/jobs/request in the ci-runners main stage.

Impact: Job polling requests from runners to the /api/v4/jobs/request route returned 5xx errors at a rate of 18.88%, exceeding the service level objective for errors in the ci-runners main stage.

Causes: A node pool upgrade caused the Redis master pod to be evicted. Because Redis was not running in high availability mode, this led to failures connecting to Redis during job polling.

Response strategy: Investigated error distribution across pods, identified Redis master pod eviction due to node pool upgrade, and confirmed the root cause. A recommendation was made to move Redis to a high availability setup to prevent similar issues.

---

This ticket was created to track INC-4072, by incident.io 🔥