# [#INC-10719: Duo Agent Platform workflows fail to start — Websocket connection closes abnormally (1006) for some flows](https://app.incident.io/gitlab/incidents/10719) Generated by Thomas Schmidt on 4 Jun 2026 05:36. All timestamps are local to Etc/UTC # Key Information <!--- Customers Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "All users with repositories on file-01 node" Requests Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "Any requests from Australia during the CloudFlare outage" Incident Severity: Copy from the incident. Use the severity at the end of the incident, not the beginning. Use the label rather than text - "~Severity::*" Start Time: Use the time when service started to be effected. Not the time the incident was opened (if they are different) End Time: Use the time when the incident was mitigated. Don't include the time in a monitoring state unless the monitoring indicated continuing problems. Total Duration: Use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations. Link to Incident Issue: Link to Incident Issue ---> | Metric | Value | |--------|-------| | Customers Affected | All customers running Duo Workflows (e.g. Code Review, Developer, custom flows) that did not have a custom `agent-config.yml` configuration that installed duo-cli via npm. | | Requests Affected | See Customers affected | | Incident Severity | Severity 2 | | Impact Start Time | Wed, 03 Jun 2026 07:00:00 UTC | | Impact End Time | Wed, 03 Jun 2026 17:42:31 UTC | | Total Duration | 10 hours, 33 minutes | | Link to Incident Issue | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22241 | # Summary **Problem**: Duo Agent Platform workflows were failing to start due to abnormal WebSocket connection closures (code 1006) when using the default agent setup (which installed the duo-cli via binary). **Impact**: New or uncustomized projects using the default Duo Agent Platform configuration could not start workflows on both GitLab.com and GDK when the duo-cli binary version was used. Mainline and custom agent setups were not affected. Early tests as of 17:25 UTC show that workflows are now starting successfully as the fix is rolled out, so remaining impact should be minimal as deployment completes. **Causes**: A recent change to the Duo Agent default configuration ([installing duo-cli via binary instead of npm](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/238715)) caused workflows to fail. The SRT sandbox's egress proxy does not support WebSocket upgrade requests, and the default agent configuration did not bypass the proxy for GitLab hosts, resulting in failed WebSocket connections. https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp/-/work_items/2445 **Response strategy**: We merged a revert to the code change that caused the issue. The fix has been deployed to canary, staging, and is now reaching production environments as of 17:25 UTC. Early verification shows workflows are starting successfully. As a temporary workaround, users could add 'setup_script: - npm install -g @gitlab/duo-cli@8.98.0' to their agent-config.yml to restore workflow startup while the rollout completes. # What went well? Use this section to highlight what went well during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared. 1. Quick discovery of the problematic MR and root cause 2. Found a workaround to unblock customers and affected colleagues # What was difficult? Use this section to highlight opportunities for improvement discovered during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared. If the improvement seems like a simplest change, consider adding it as a corrective action above instead. Think about how to improve response next time, and consider any patterns pointing to broader issues, like “key person risk.” 1. The existing monitoring and logging systems did not clearly indicate a problem because of the nature of the bug (i.e. it would not reach out logging / monitoring layer). We should do improvements here. 2. The deploy of the fix took a long time until it ran through all pipelines # Investigation Details <details> <summary>Timeline</summary> # Incident Timeline **2026-06-03** **07:00:00** **Impact started at** Custom timestamp "Impact started at" occurred **07:09:02** **Incident reported by Thomas Schmidt** Thomas Schmidt reported the incidentSeverity: Severity 2Status: Investigating **07:09:07** **Escalated to GitLab.com Production** Your workflow manually escalated the incident to the escalation path [GitLab.com Production](https://app.incident.io/gitlab/on-call/escalation-paths/01K5YWH36WZZX2YT9HNRR0WMB0) **07:09:08** **Escalated to Incident Manager Oncall (IMOC)** Your workflow manually escalated the incident to the escalation path [Incident Manager Oncall (IMOC)](https://app.incident.io/gitlab/on-call/escalation-paths/01K8VN9TTAJH6N3WKNCAMNXQ40) **07:09:37** **Escalation acknowledged by Patrick Bajao** Patrick Bajao acknowledged the escalation **07:09:43** **Escalation acknowledged by Vladimir Glafirov** Vladimir Glafirov acknowledged the escalation **07:12:35** **Message from Thomas Schmidt** Thomas Schmidt's message was pinned by Aaron Richter Ok bare with me, this is the first incident I am creating :smile: A couple of infos up front from my local agent assisted investigation: - DAP ambient flows (e.g. issue/MR mention → agent) fail whenever the `srt` sandbox is active. - All ordinary HTTP/GraphQL calls succeed (200) — only the WebSocket is affected, so the failure is silent until the workflow tries to connect. Root cause (hypothesis, with reproduction): - When the `srt` sandbox starts, it routes egress through a proxy (`localhost:3128`). That proxy does not relay the WebSocket `101 Switching Protocols` upgrade, so the workflow WS connection dies. - The GitLab instance host is not in `srt`s `NO_PROXY`, so the WS is sent through the proxy. - Not a CLI version issue (8.98 and 8.99 both fail; a prod job on 8.99 with the sandbox inactive worked). Why now: - The bug was masked whenever the sandbox failed to start (job ran directly → no proxy → WS worked). Recent changes made the sandbox start in more jobs — notably the 2026‑06‑01 move of the sandbox dir `/opt` →`/var/tmp` (MR (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/238285)), which makes it succeed on non-root/hardened images that previously fell back to direct execution. Evidence: - Reproduced deterministically in `workflow-generic-image:v0.0.6` with a local WS server (no GitLab/auth): direct → `101` :white_check_mark:; through `srt` proxy → TIMEOUT :x: (HTTP still `200`). Repro scripts available. - Confirmed across GDK jobs and gitlab.com (http://gitlab.com) prod jobs (tokinator). Proposed mitigation (TBD): - Add the GitLab instance host / sandbox `allowedDomains` to `NO_PROXY` so the WS bypasses the proxy, and/or make the `srt` proxy support WebSocket upgrades. **07:26:12** **Message from Thomas Schmidt** Thomas Schmidt's message was pinned by Aaron RichterThere is also a couple of logs that folks shared where they bumped into the problem: - https://gitlab.com/gitlab-com/gl-security/product-security/psirt-group/tokinator/-/automate/agent-sessions/4120312 - https://gitlab.com/gitlab-com/gl-security/product-security/psirt-group/tokinator/-/automate/agent-sessions/4120275 **07:27:18** **Incident escalated** Someone manually escalated the incident **07:27:26** **Escalation acknowledged by Vladimir Glafirov** Vladimir Glafirov acknowledged the escalation **07:28:44** **Escalated to tier2 - AI Escalation** Vladimir Glafirov manually escalated the incident to the escalation path [tier2 - AI Escalation](https://app.incident.io/gitlab/on-call/escalation-paths/01K22CAST6CK8Y4DVN7ET8YQZX) **07:44:23** **Escalation acknowledged by Patrick Bajao** Patrick Bajao acknowledged the escalation **08:12:34** **Severity downgraded from Severity 2 → Severity 3** Vladimir Glafirov shared an update Severity: ~~Severity 2~~ → Severity 3 We have confirmed that Duo Agent Platform workflows fail to start with WebSocket connection errors (code 1006) when using the default agent configuration and sandboxing is active. This is reproducible both locally and in production on GitLab.com projects without custom agent config.Dashboards for duo-workflow-svc show no significant increase in error ratios or drop in traffic, which suggests the failures are not captured as server-side errors. However, runner-side connection failures are consistently observed in job logs for default setups.Custom agent configurations and certain mainline projects appear unaffected, but new or uncustomized projects using the default image are consistently impacted. There is ongoing discussion about whether this warrants Severity 2, given the lack of visible service-wide impact in dashboard metrics. **08:53:18** **Update shared** Vladimir Glafirov (via @incident) shared an updateThe incident has been downgraded to Severity 3. We confirmed that Duo Agent Platform workflows fail to start with WebSocket connection errors (code 1006) when the default agent configuration is used with sandboxing enabled. This impacts new or uncustomized projects, while custom agent configurations remain unaffected.Service dashboards for duo-workflow-svc show no significant increase in error rates or drop in traffic, indicating that server-side errors are not being recorded for these failures. Instead, runner-side connection failures are visible in job logs for affected setups. A revert merge request has been opened to address the suspected cause: [Revert "Merge branch 'duo/fix/600436-migrate-duo-cli-npm-to-binary-fixes' into 'master'"](#239041)(https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239041). Deployment of this revert may be delayed due to scheduled auto-deploy maintenance. **09:56:25** **Update shared** Thomas Schmidt shared an update The revert merge request ([MR #239041](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239041)) addressing the workflow startup failures has been merged. This change rolls back the Duo CLI binary installation for default agent configurations, which was found to cause abnormal WebSocket closure errors (code 1006) when sandboxing is enabled. We are now awaiting the deployment of this fix. Deployment is was delayed due to scheduled auto-deploy maintenance (which has finished by now)Impact remains limited to new or uncustomized projects using the default agent setup with sandboxing; mainline and custom agent configurations are unaffected. There is no service-wide error spike or traffic drop visible in dashboards. Monitoring will continue until the fix has been deployed and verified. **10:47:22** **Update shared** Vladimir Glafirov (via @incident) shared an update The revert merge request ([MR #239041](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239041)) addressing Duo Agent Platform workflow startup failures has been merged. The fix rolls back the recent Duo CLI binary installation change, which caused abnormal WebSocket closure errors (code 1006) for default agent configurations with sandboxing enabled. Deployment of the fix is still pending, as auto-deploy maintenance delayed rollout. Impact remains limited to new or uncustomized projects using the default agent setup with sandboxing; mainline and custom agent configurations are not affected. There is still no error spike or traffic drop visible in service dashboards. As a temporary workaround, users can add the following to their `agent-config.yml` to restore workflow startup: ``` setup_script: - npm install -g @gitlab/duo-cli@8.98.0 ``` We are monitoring for deployment completion and will verify resolution once the fix is live. **11:35:58** **Status changed from Investigating → Fixing** Thomas Schmidt shared an update Status: ~~Investigating~~ → Fixing The fix for the Duo Agent Platform workflow failures has been merged and is included in the auto-deploy branch, but deployment to gitlab.com is delayed due to ongoing auto-deploy maintenance.Users with new or uncustomized projects using the default agent setup and sandboxing remain impacted until the deployment completes. There is no visible service-wide error spike or traffic drop in dashboards.Progress on deployment can be tracked in the #announcements channel or by following the auto-deploy branch/version labels on the relevant merge requests. Verification of the fix will occur once the deployment is live. **11:35:58** **Identified at** Custom timestamp "Identified at" occurred **13:32:04** **Update shared** Thomas Schmidt shared an updateThe revert MR addressing the Duo Agent Platform workflow failures is now being deployed to the canary environment. The auto-deploy pipeline has started for gprd-cny, and we are monitoring its progress.Once the deployment completes baking in canary, the fix will be promoted to staging and production environments. We will continue to track deployment status and verify resolution once the fix reaches production. **14:00:06** **Severity upgraded from Severity 3 → Severity 2** Thomas Schmidt shared an update Severity: ~~Severity 3~~ → Severity 2 **15:18:23** **Update shared** Aaron Richter (via @incident) shared an updateThe fix for the Duo Agent Platform workflow failures has progressed to the canary environment, and initial tests show that at least some affected flows are now starting successfully.Production rollout is ongoing. Users on the default agent setup with sandboxing should see gradual improvement as the deployment completes. We continue to monitor workflow startups and will verify full resolution as soon as the fix is live in production.If you continue to experience failures, please check the workflow environment to confirm if the fix has reached your jobs. **16:18:32** **Update shared** Aaron Richter (via @incident) shared an updateThe fix for the Duo Agent Platform flow startup failures is still rolling out to production. Early tests indicate that some affected flows are now starting successfully in the canary environment.Users with new or uncustomized projects using the default agent setup and sandboxing should see gradual improvement as the deployment progresses. Workarounds remain available for those still impacted. We continue to monitor the rollout and will confirm full resolution once the fix is live across all environments. **17:25:16** **Message from Thomas Schmidt** Thomas Schmidt's message was pinned by Aaron Richter Nice. Thanks @thedrummeraki. I did some early tests and it looks like it fixes the issue https://gitlab.slack.com/archives/C0B7RDHPMV1/p1780506550088099 Since it only hit production now, I am a tiny bit confused but maybe I actually hit some pods that already had the fix when I was testing. **17:42:31** **Fixed at** Custom timestamp "Fixed at" occurred **17:42:31** **Incident resolved and entered the post-incident flow** Aaron Richter shared an update Status: ~~Fixing~~ → Documenting # Investigation Notes </details> Any details you may want to add about the investigation can go here. # Lessons Learned ### General lessons _What did we learn from this incident that we want to carry forward?_ 1. The problem should have been able to be identified with proper local end 2 end tests. There were a few circumstances that might have masked it 1. Developers testing the MR might have had a local setup with customization (agent-config file) that would have masked the problem (e.g. no SRT active) We need to prioritise https://gitlab.com/gitlab-org/gitlab/-/work_items/474195 and https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp/-/work_items/1891 to catch these issues in an automated way. ### What can we do to reduce detection time by half? _Consider: alerting gaps, missing metrics, lack of dashboards, unclear ownership, delayed escalation paths._ 1. Introduce metrics that track flow failures for duo cli webhook establishing and similar problems -> https://gitlab.com/gitlab-org/gitlab/-/work_items/602178 ### What can we do to reduce resolution time by half? _Consider: missing runbooks, unclear mitigation steps, tooling gaps, communication delays, lack of automation._ 1. Faster end to end deploy process # Review Guidelines This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI. ### For the person opening the Incident Review - [x] Set the title to `Incident Review: (Incident issue name)` - [x] Assign a `Service::*` label (most likely matching the one on the incident issue) - [x] Set a `Severity::*` label which matches the incident - [x] In the `Key Information` section, make sure to include a link to the incident issue - [x] Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) **The DRI for the incident review is the issue assignee.** ### For the assigned DRI - [x] Fill in the remaining fields in the `Key Information` section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - [x] If there are metrics showing `Customers Affected` or `Requests Affected`, link those metrics in those fields - [x] For all S1 and S2 incidents, begin the [Feature Change Lock (FCL) process](https://handbook.gitlab.com/handbook/engineering/#feature-change-locks) and [open an issue in the FCL project](https://gitlab.com/gitlab-com/feature-change-locks/-/issues/new?description_template=feature-change-lock). - [x] Create a few short sentences in the Summary section summarizing what happened (TL;DR) - [x] Link any corrective actions and describe any other actions or outcomes from the incident - [x] Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported? - [x] Once discussion wraps up in the comments, summarize any takeaways in the details section - [x] If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue. - [x] S1 incidents [require a public RCA within 7 days](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/incident-review/#timeline-that-we-expect-for-reviews-to-be-completed) of the incident. If this review cannot be made public, [create a separate public RCA](https://handbook.gitlab.com/handbook/engineering/root-cause-analysis/#how-to-perform-an-rca). - [x] Close the review before the due date - [x] Go back to the incident channel or page and close out the remaining post-incident tasks