# [INC-7055: Redis deployment error causes delayed CI processing and GitLab Duo outage](https://app.incident.io/gitlab/incidents/7055) Generated by John Jarvis on 4 Feb 2026 14:33. All timestamps are local to Etc/UTC # Key Information <!--- Customers Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "All users with repositories on file-01 node" Requests Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "Any requests from Australia during the CloudFlare outage" Incident Severity: Copy from the incident. Use the severity at the end of the incident, not the beginning. Use the label rather than text - "~Severity::*" Start Time: Use the time when service started to be effected. Not the time the incident was opened (if they are different) End Time: Use the time when the incident was mitigated. Don't include the time in a monitoring state unless the monitoring indicated continuing problems. Total Duration: Use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations. Link to Incident Issue: Link to Incident Issue ---> | Metric | Value | | ------ | ------ | | Customers Affected | At least 53,726 unique namespaces, CI-wise ([Kibana](https://log.gprd.gitlab.net/app/r/s/0eZ7U)), 1266 unique Duo workflows ([Kibana](https://log.gprd.gitlab.net/app/r/s/Z8r5P))| | Requests Affected | 24,543,552 ([Workhorse](https://log.gprd.gitlab.net/app/r/s/YTbyw), [Rails](https://log.gprd.gitlab.net/app/r/s/KpbSe)) | | Incident Severity | Severity 1 | | Impact Start Time | Tue, 03 Feb 2026 12:30:00 UTC| | Impact End Time | Tue, 03 Feb 2026 17:44:00 UTC | | Total Duration | 4 hours, 53 minutes | | Link to Incident Issue | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21192 | # Summary **Problem**: Loss of connectivity to a production Redis cluster caused CI job processing delays and a complete outage of GitLab Duo features. Although Redis service was restored, the CI job queue is rising again due to unresolved processing issues. **Impact**: CI job delays prevented pipelines from completing. All users were unable to access GitLab Duo features in both the VSCode extension and web interface, resulting in a complete outage for this functionality. Deployments and feature flag changes are blocked. After restoring Redis, GitLab Duo became available and CI pipelines started recovering, but the CI job backlog grew again, causing ongoing delays for CI jobs. Deployments and feature flag changes remain paused until the CI queue stabilizes. **Causes**: A [configuration change to Redis increased CPU reservations](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/675/diffs) on Sentinel nodes resulted in a pod being un-schedulable due to there not being enough cpu capacity on the node. Following that an operator initiated scaling event was done which in turn triggered an eviction event. The PDB (pod disruption budget) correctly blocked the pod eviction, but GKE's load balancer controller removed the NEG (network endpoint group) because NEGs are removed independently of PDB. The pod stayed running (PDB worked), but became unreachable via the ILB . **Response strategy**: We rolled back the Redis configuration change to restore service. GitLab Duo and CI pipelines recovered, but the CI job backlog began growing again. Engineers are investigating Sidekiq processing as a possible bottleneck. Deployments and feature flag changes remain paused until the CI queue recovers. # What went well? 1. The origin of the degradation was identified quickly. 2. A revert of the mis-configuration was quickly created. 3. There was a wealth of K8s experience on the incident call that identified why the new configuration wasn't working as expected. # What was difficult? 1. The mis-configuration revert wasn't able to be merged because pipelines couldn't be processed (due to the incident itself). 2. ArgoCD access wasn't granted in advance to all SREs, and the approval process was lengthy. 3. At some point during recovery, our existing infrastructure wasn't able to keep up with the backed up CI jobs that we started backing up once more. # Investigation Details <details> <summary> Timeline</summary> # Incident Timeline **2026-02-03** **12:30:00** **Impact started at** Custom timestamp "Impact started at" occurred **12:51:06** **Incident reported by Dat Tang** Dat Tang reported the incident Severity: Severity 3 Status: Investigating **12:51:59** **Message from Dat Tang** Dat Tang pinned their own message Original report https://gitlab.slack.com/archives/CETG54GQ0/p1770122163985369 **12:53:28** **Message from Dat Tang** Dat Tang pinned their own message https://gitlab.slack.com/archives/CETG54GQ0/p1770123087891879?thread_ts=1770123034.349669&cid=CETG54GQ0 **12:53:41** **Message from Falko Sieverding** Falko Sieverding's message was pinned by Dat Tang Happening for me in the Webui **12:56:22** **Message from Falko Sieverding** Falko Sieverding's message was pinned by Dat Tang There was a custome rincident with the same message: https://gitlab.slack.com/archives/C08TUAH45NG/p1769759884493659 **12:56:33** **Severity upgraded from Severity 3 → Severity 1** Dat Tang shared an update Severity: ~~Severity 3~~ → Severity 1 **12:56:34** **Escalated to GitLab.com Production** Your workflow manually escalated the incident to the escalation path [GitLab.com Production](https://app.incident.io/gitlab/on-call/escalation-paths/01K5YWH36WZZX2YT9HNRR0WMB0) **12:56:34** **Escalated to Incident Manager Oncall (IMOC)** Your workflow manually escalated the incident to the escalation path [Incident Manager Oncall (IMOC)](https://app.incident.io/gitlab/on-call/escalation-paths/01K8VN9TTAJH6N3WKNCAMNXQ40) **12:58:09** **Escalated to tier2 - AI** Donna Alexandra manually escalated the incident to the escalation path [tier2 - AI Escalation](https://app.incident.io/gitlab/on-call/escalation-paths/01K22CAST6CK8Y4DVN7ET8YQZX) **12:58:19** **Update shared** Abhinaba Ghosh (via @incident) shared an update GitLab Duo is unavailable for all users in both the VSCode extension and the web interface. Attempting to use the agentic chat triggers an error: "GitLab Duo is already responding to this chat in another tab or location." This issue is blocking deployments and feature flag changes. The problem is persistent across multiple environments and has been confirmed by several users. There is no indication that a recent deployment caused the issue, as the last production deployment was approximately six hours prior to the first report. A previous similar incident affected a self-managed customer using GitLab version 18.8.2, where misconfigured Redis connections prevented chat sessions from working. However, the root cause for this incident has not yet been determined. Deployments have been blocked as a precaution while the investigation continues. **13:07:03** **Message from Igor Drozdov** Igor Drozdov's message was pinned by Donna Alexandra https://log.gprd.gitlab.net/app/r/s/WWhYn **13:17:52** **Update shared** Sebastian Rehm shared an update The underlying reason is that workhorse cannot connect to Redis, which means DAP cannot acquire a lock for the session and fails. **13:19:22** **Image posted by John Jarvis** John Jarvis posted an image to the channel Reds pubsub requests dropped to zero https://dashboards.gitlab.net/goto/afc4osuaxjfuoc?orgId=1  **13:59:51** **Message from Dat Tang** Dat Tang pinned their own message From the current spec of `redis-pubsub-node-2` , there are 4 containers, requesting 4+2+0.5+0.5=7 CPU. There are only around 6.4 available in the node -> cannot schedule **14:00:17** **Message from Dat Tang** Dat Tang pinned their own message Previously we requested 4+1+0.5+0.5=6 CPU, so just enough **14:48:57** **Image posted by Rehab Hassanein** Rehab Hassanein posted an image to the channel It's worth noting that the shards mostly affected by this are `small` , `private` and `gitlab-org` - the rest of the shards have pretty much processed everything in their queue (around `14:18 UTC`): ref: https://dashboards.gitlab.net/goto/efc4wrse4yyo0a?orgId=1  **15:11:00** **Message from Igor Wiedler** Igor Wiedler's message was pinned by Dat Tang @Dat Tang re: pod sizing, I suspect a difference in daemonsets between gstg and gprd may have been why this wasn't caught on gstg **15:12:00** **Identified at** Custom timestamp "Identified at" occurred **15:12:31** **Status changed from Investigating → Fixing** Dat Tang shared an update Status: ~~Investigating~~ → Fixing **15:50:19** **Update shared** John Jarvis (via @incident) shared an update Service to the affected Redis cluster has been restored by reverting the configuration change that increased CPU reservations on Sentinel nodes. This resolved the outage and restored connectivity for dependent services, including GitLab Duo and CI job processing. GitLab Duo features in both the VSCode extension and web interface are now available again. CI pipelines are recovering as the backlog of queued jobs is processed. Apdex scores and PgBouncer saturation are improving, but users may still experience some delays while the backlog clears. Deployments and feature flag changes remain paused until the CI queue returns to normal levels. We are monitoring recovery progress and will keep the incident open until queues have cleared. All related alerts have been resolved. For additional details, see the incident tracker: [work item 21192](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21192), and status page: [GitLab Duo incident](https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/6981f3ea6a29f904fc59b66e). **16:30:04** **Update shared** Adil Farrukh shared an update After restoring Redis, CI pipelines and GitLab Duo recovered, but the CI job backlog—which had previously dropped—has begun rising again. The latest dashboards show an increase in pending CI jobs, that is being investigated with Sidekiq not auto-scaling as much as needed. The Redis outage itself is resolved, and GitLab Duo remains available. However, the underlying issue with CI job processing is not fully cleared, and the backlog is not draining as expected. Engineers are investigating whether this is related to delayed Sidekiq processing or other factors affecting job throughput. No changes have been made to deployments or feature flags, which remain paused until the CI queue stabilizes. Updates will be provided as soon as the cause of the renewed backlog growth is confirmed. **17:31:05** **Update shared** Adil Farrukh (via @incident) shared an update The root Redis outage has been resolved and GitLab Duo features remain fully available. However, CI job processing is still affected: after initial recovery, the job backlog began increasing again. This is causing ongoing delays for CI pipelines and keeping deployments and feature flag changes on hold. Engineering teams have confirmed the Redis issue is resolved and have shifted focus to investigating Sidekiq processing and job throughput as likely causes for the renewed backlog growth. The current priority is diagnosing and mitigating this bottleneck so that CI jobs can process normally and the queue can drain. We are not resuming deployments or feature flag changes until CI processing is stable and delays have cleared. For more details, refer to the [incident tracker work item](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21192) and the [GitLab Duo status page](https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/6981f3ea6a29f904fc59b66e). **17:44:00** **Fixed at** Custom timestamp "Fixed at" occurred **17:44:23** **Incident resolved and entered the post-incident flow** Adil Farrukh shared an update Status: ~~Fixing~~ → Documenting # Investigation Notes </details> Any details you may want to add about the investigation can go here. # Follow-ups All followups are tracked in https://gitlab.com/groups/gitlab-com/gl-infra/tenant-scale/tenant-services/-/work_items/50 ### Duo related corrective actions 1. [Increase server SLI error threshold from 0.95 to 0.99](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/1890): This addresses how we didn't get alerted for the spike in errors 1. [Ignore misconfigured Redis for DWS locking](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/221897): This will prevent the Duo failure altogether for a similar Redis issue since it does not need to be fatal 1. [Add SLI + alert in workhorse to monitor Duo Agent Platform session locking errors](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/1886): Even though it is no longer fatal, we can create a more specific alert to for lock acquisition failures. ### Redis infrastructure related corrective actions 1. [Add warnings for future changes to requests on redis-pubsub](https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/merge_requests/774) 1. [Migrate redis-pubsub to a VM-based cluster (name it redis-workhorse)](https://gitlab.com/groups/gitlab-com/gl-infra/tenant-scale/tenant-services/-/work_items/51): We are no longer investing in Redis on K8s and should migrate this workload, however it will need to be scheduled as a larger project. 1. [Implement pause mechanism for ArgoCD applications](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28352) # Review Guidelines This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI. ### For the person opening the Incident Review - [x] Set the title to `Incident Review: (Incident issue name)` - [x] Assign a `Service::*` label (most likely matching the one on the incident issue) - [x] Set a `Severity::*` label which matches the incident - [x] In the `Key Information` section, make sure to include a link to the incident issue - [ ] Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) **The DRI for the incident review is the issue assignee.** ### For the assigned DRI - [x] Fill in the remaining fields in the `Key Information` section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - [x] If there are metrics showing `Customers Affected` or `Requests Affected`, link those metrics in those fields - [x] For all S1 and S2 incidents, begin the [Feature Change Lock (FCL) process](https://handbook.gitlab.com/handbook/engineering/#feature-change-locks) and [open an issue in the FCL project](https://gitlab.com/gitlab-com/feature-change-locks/-/issues/new?description_template=feature-change-lock). - [x] Create a few short sentences in the Summary section summarizing what happened (TL;DR) - [x] Link any corrective actions and describe any other actions or outcomes from the incident - [x] Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported? - [ ] Once discussion wraps up in the comments, summarize any takeaways in the details section - [ ] If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue. - [ ] Close the review before the due date - [ ] Go back to the incident channel or page and close out the remaining post-incident tasks