Incident Review: Redis deployment error causes delayed CI processing and GitLab Duo outage
INC-7055: Redis deployment error causes delayed CI processing and GitLab Duo outage
Generated by John Jarvis on 4 Feb 2026 14:33. All timestamps are local to Etc/UTC
Key Information
| Metric | Value |
|---|---|
| Customers Affected | At least 53,726 unique namespaces, CI-wise (Kibana), 1266 unique Duo workflows (Kibana) |
| Requests Affected | 24,543,552 (Workhorse, Rails) |
| Incident Severity | Severity 1 |
| Impact Start Time | Tue, 03 Feb 2026 12:30:00 UTC |
| Impact End Time | Tue, 03 Feb 2026 17:44:00 UTC |
| Total Duration | 4 hours, 53 minutes |
| Link to Incident Issue | #21192 (closed) |
Summary
Problem: Loss of connectivity to a production Redis cluster caused CI job processing delays and a complete outage of GitLab Duo features. Although Redis service was restored, the CI job queue is rising again due to unresolved processing issues.
Impact: CI job delays prevented pipelines from completing. All users were unable to access GitLab Duo features in both the VSCode extension and web interface, resulting in a complete outage for this functionality. Deployments and feature flag changes are blocked. After restoring Redis, GitLab Duo became available and CI pipelines started recovering, but the CI job backlog grew again, causing ongoing delays for CI jobs. Deployments and feature flag changes remain paused until the CI queue stabilizes.
Causes: A configuration change to Redis increased CPU reservations on Sentinel nodes resulted in a pod being un-schedulable due to there not being enough cpu capacity on the node. Following that an operator initiated scaling event was done which in turn triggered an eviction event. The PDB (pod disruption budget) correctly blocked the pod eviction, but GKE's load balancer controller removed the NEG (network endpoint group) because NEGs are removed independently of PDB. The pod stayed running (PDB worked), but became unreachable via the ILB .
Response strategy: We rolled back the Redis configuration change to restore service. GitLab Duo and CI pipelines recovered, but the CI job backlog began growing again. Engineers are investigating Sidekiq processing as a possible bottleneck. Deployments and feature flag changes remain paused until the CI queue recovers.
What went well?
- The origin of the degradation was identified quickly.
- A revert of the mis-configuration was quickly created.
- There was a wealth of K8s experience on the incident call that identified why the new configuration wasn't working as expected.
What was difficult?
- The mis-configuration revert wasn't able to be merged because pipelines couldn't be processed (due to the incident itself).
- ArgoCD access wasn't granted in advance to all SREs, and the approval process was lengthy.
- At some point during recovery, our existing infrastructure wasn't able to keep up with the backed up CI jobs that we started backing up once more.
Investigation Details
Timeline
Incident Timeline
2026-02-03
12:30:00 Impact started at
Custom timestamp "Impact started at" occurred
12:51:06 Incident reported by Dat Tang
Dat Tang reported the incident
Severity: Severity 3
Status: Investigating
12:51:59 Message from Dat Tang
Dat Tang pinned their own message
Original report https://gitlab.slack.com/archives/CETG54GQ0/p1770122163985369
12:53:28 Message from Dat Tang
Dat Tang pinned their own message
12:53:41 Message from Falko Sieverding
Falko Sieverding's message was pinned by Dat Tang
Happening for me in the Webui
12:56:22 Message from Falko Sieverding
Falko Sieverding's message was pinned by Dat Tang
There was a custome rincident with the same message: https://gitlab.slack.com/archives/C08TUAH45NG/p1769759884493659
12:56:33 Severity upgraded from Severity 3 → Severity 1
Dat Tang shared an update
Severity: Severity 3 → Severity 1
12:56:34 Escalated to GitLab.com Production
Your workflow manually escalated the incident to the escalation path GitLab.com Production
12:56:34 Escalated to Incident Manager Oncall (IMOC)
Your workflow manually escalated the incident to the escalation path Incident Manager Oncall (IMOC)
12:58:09 Escalated to tier2 - AI
Donna Alexandra manually escalated the incident to the escalation path tier2 - AI Escalation
12:58:19 Update shared
Abhinaba Ghosh (via @incident) shared an update
GitLab Duo is unavailable for all users in both the VSCode extension and the web interface. Attempting to use the agentic chat triggers an error: "GitLab Duo is already responding to this chat in another tab or location." This issue is blocking deployments and feature flag changes.
The problem is persistent across multiple environments and has been confirmed by several users. There is no indication that a recent deployment caused the issue, as the last production deployment was approximately six hours prior to the first report.
A previous similar incident affected a self-managed customer using GitLab version 18.8.2, where misconfigured Redis connections prevented chat sessions from working. However, the root cause for this incident has not yet been determined.
Deployments have been blocked as a precaution while the investigation continues.
13:07:03 Message from Igor Drozdov
Igor Drozdov's message was pinned by Donna Alexandra
https://log.gprd.gitlab.net/app/r/s/WWhYn
13:17:52 Update shared
Sebastian Rehm shared an update
The underlying reason is that workhorse cannot connect to Redis, which means DAP cannot acquire a lock for the session and fails.
13:19:22 Image posted by John Jarvis
John Jarvis posted an image to the channel
Reds pubsub requests dropped to zero https://dashboards.gitlab.net/goto/afc4osuaxjfuoc?orgId=1
13:59:51 Message from Dat Tang
Dat Tang pinned their own message
From the current spec of redis-pubsub-node-2 , there are 4 containers, requesting 4+2+0.5+0.5=7 CPU. There are only around 6.4 available in the node -> cannot schedule
14:00:17 Message from Dat Tang
Dat Tang pinned their own message
Previously we requested 4+1+0.5+0.5=6 CPU, so just enough
14:48:57 Image posted by Rehab Hassanein
Rehab Hassanein posted an image to the channel
It's worth noting that the shards mostly affected by this are small , private and gitlab-org - the rest of the shards have pretty much processed everything in their queue (around 14:18 UTC):
ref: https://dashboards.gitlab.net/goto/efc4wrse4yyo0a?orgId=1
15:11:00 Message from Igor Wiedler
Igor Wiedler's message was pinned by Dat Tang
@Dat Tang re: pod sizing, I suspect a difference in daemonsets between gstg and gprd may have been why this wasn't caught on gstg
15:12:00 Identified at
Custom timestamp "Identified at" occurred
15:12:31 Status changed from Investigating → Fixing
Dat Tang shared an update
Status: Investigating → Fixing
15:50:19 Update shared
John Jarvis (via @incident) shared an update
Service to the affected Redis cluster has been restored by reverting the configuration change that increased CPU reservations on Sentinel nodes. This resolved the outage and restored connectivity for dependent services, including GitLab Duo and CI job processing.
GitLab Duo features in both the VSCode extension and web interface are now available again. CI pipelines are recovering as the backlog of queued jobs is processed. Apdex scores and PgBouncer saturation are improving, but users may still experience some delays while the backlog clears. Deployments and feature flag changes remain paused until the CI queue returns to normal levels.
We are monitoring recovery progress and will keep the incident open until queues have cleared. All related alerts have been resolved. For additional details, see the incident tracker: work item 21192, and status page: GitLab Duo incident.
16:30:04 Update shared
Adil Farrukh shared an update
After restoring Redis, CI pipelines and GitLab Duo recovered, but the CI job backlog—which had previously dropped—has begun rising again. The latest dashboards show an increase in pending CI jobs, that is being investigated with Sidekiq not auto-scaling as much as needed.
The Redis outage itself is resolved, and GitLab Duo remains available. However, the underlying issue with CI job processing is not fully cleared, and the backlog is not draining as expected. Engineers are investigating whether this is related to delayed Sidekiq processing or other factors affecting job throughput.
No changes have been made to deployments or feature flags, which remain paused until the CI queue stabilizes. Updates will be provided as soon as the cause of the renewed backlog growth is confirmed.
17:31:05 Update shared
Adil Farrukh (via @incident) shared an update
The root Redis outage has been resolved and GitLab Duo features remain fully available. However, CI job processing is still affected: after initial recovery, the job backlog began increasing again. This is causing ongoing delays for CI pipelines and keeping deployments and feature flag changes on hold.
Engineering teams have confirmed the Redis issue is resolved and have shifted focus to investigating Sidekiq processing and job throughput as likely causes for the renewed backlog growth. The current priority is diagnosing and mitigating this bottleneck so that CI jobs can process normally and the queue can drain.
We are not resuming deployments or feature flag changes until CI processing is stable and delays have cleared. For more details, refer to the incident tracker work item and the GitLab Duo status page.
17:44:00 Fixed at
Custom timestamp "Fixed at" occurred
17:44:23 Incident resolved and entered the post-incident flow
Adil Farrukh shared an update
Status: Fixing → Documenting
Investigation Notes
Follow-ups
All followups are tracked in gitlab-com/gl-infra/tenant-scale/tenant-services#50 (closed)
Duo related corrective actions
- Increase server SLI error threshold from 0.95 to 0.99: This addresses how we didn't get alerted for the spike in errors
- Ignore misconfigured Redis for DWS locking: This will prevent the Duo failure altogether for a similar Redis issue since it does not need to be fatal
- Add SLI + alert in workhorse to monitor Duo Agent Platform session locking errors: Even though it is no longer fatal, we can create a more specific alert to for lock acquisition failures.
Redis infrastructure related corrective actions
- Add warnings for future changes to requests on redis-pubsub
- Migrate redis-pubsub to a VM-based cluster (name it redis-workhorse): We are no longer investing in Redis on K8s and should migrate this workload, however it will need to be scheduled as a larger project.
- Implement pause mechanism for ArgoCD applications
Review Guidelines
This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI.
For the person opening the Incident Review
- Set the title to
Incident Review: (Incident issue name) - Assign a
Service::*label (most likely matching the one on the incident issue) - Set a
Severity::*label which matches the incident - In the
Key Informationsection, make sure to include a link to the incident issue - Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) The DRI for the incident review is the issue assignee.
For the assigned DRI
- Fill in the remaining fields in the
Key Informationsection, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - If there are metrics showing
Customers AffectedorRequests Affected, link those metrics in those fields - For all S1 and S2 incidents, begin the Feature Change Lock (FCL) process and open an issue in the FCL project.
- Create a few short sentences in the Summary section summarizing what happened (TL;DR)
- Link any corrective actions and describe any other actions or outcomes from the incident
- Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported?
- Once discussion wraps up in the comments, summarize any takeaways in the details section
- If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue.
- Close the review before the due date
- Go back to the incident channel or page and close out the remaining post-incident tasks