2026-02-03: Redis deployment error causes delayed CI processing and GitLab Duo outage

Redis deployment error causes delayed CI processing and GitLab Duo outage (Severity 1)

Problem: Loss of connectivity to a production Redis cluster caused CI job processing delays and a complete outage of GitLab Duo features. Although Redis service was restored, the CI job queue is rising again due to unresolved processing issues.

Impact: CI job delays prevented pipelines from completing. All users were unable to access GitLab Duo features in both the VSCode extension and web interface, resulting in a complete outage for this functionality. Deployments and feature flag changes are blocked. After restoring Redis, GitLab Duo became available and CI pipelines started recovering, but the CI job backlog grew again, causing ongoing delays for CI jobs. Deployments and feature flag changes remain paused until the CI queue stabilizes.

Causes: A configuration change to Redis increased CPU reservations on Sentinel nodes, causing a Redis outage that prevented workhorse from connecting to Redis.

Response strategy: We rolled back the Redis configuration change to restore service. GitLab Duo and CI pipelines recovered, but the CI job backlog began growing again. Engineers are investigating Sidekiq processing as a possible bottleneck. Deployments and feature flag changes remain paused until the CI queue recovers.

This ticket was created to track INC-7055, by incident.io 🔥