<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/engineering/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=630749) </details> <!--IssueSummary end--> ## Summary When a Workhorse-run MCP tool call fails at the protocol level, Workhorse ends the whole DAP session instead of returning an error result to the LLM. DWS then cannot update the session status, so the chat stays in "running" and the UI shows "producing an answer" with no end. ## Evidence (gitlab.com, 2026-09-24) `orbit_agent/v1` sessions 8647941, 8648386 and 8648694 were stuck for 10 to 38 minutes. They ended only when a DWS deploy killed them (`DWS_SERVER_SHUTDOWN`). In all three sessions: 1. DWS sends `RunMCPTool orbit_invoke_command` (an Orbit `query_graph`). 2. 30.3 to 30.4 s later, DWS logs `Client-side streaming has been closed.` and then `Aborting workflow...`. 3. At the same second, the Orbit server logs ClickHouse `TIMEOUT_EXCEEDED` (30000 ms). 4. DWS tries to send `PATCH status_event=drop` through the closed stream. Nothing reaches Rails. Two more sessions (8648848, 8648888) closed about 0.5 s after an `orbit_invoke_command` call, with no Orbit error. The cause is probably the same code path (see "Other triggers"), but I did not confirm it because I have no access to the Workhorse logs. ## Root cause ```mermaid sequenceDiagram participant B as Browser participant W as Workhorse participant R as Rails participant O as Orbit + ClickHouse participant D as DWS Note over B,D: Browser to Workhorse WebSocket, Workhorse to DWS gRPC stream D->>W: RunMCPTool (Orbit query) W->>R: POST /api/v4/orbit/mcp R-->>W: 200 + Send-Data (TimeoutSeconds 30) W->>O: ExecuteQuery (stream deadline 30 s) Note over O: ClickHouse query runs (max_execution_time 30 s) Note over W: The 30 s stream deadline fires first Note over W: handleRecvError writes a plain HTTP 504, not an MCP isError result.<br/>CallTool() returns an error, and handleAgentAction() returns it.<br/>The runner ends the workflow. W-xB: close WebSocket W-xD: close gRPC stream O--xW: TIMEOUT_EXCEEDED (nobody reads it) D--xW: PATCH status_event=drop (never reaches Rails) Note over R: Session status stays "running" Note over B: "producing an answer" with no end Note over D: RPC waits until the next DWS deploy ``` - `workhorse/internal/ai_assist/duoworkflow/runner.go` `handleAgentAction`: when `mcpManager.CallTool` returns an error, the function returns that error. The runner then ends the workflow and closes both the WebSocket and the DWS stream. - DWS sends its last status update through the same stream (executor outbox). When the stream is closed, the update is lost and Rails keeps the session in `running`. - For `RunHTTPRequest` actions and for unsupported client actions, Workhorse already sends a failed `ActionResponse` back to DWS. MCP tool calls are the only case that ends the session. - `workhorse/internal/orbit/sendquery.go`: when an Orbit query fails in Workhorse (stream deadline, stream error, redaction failure), MCP callers get a plain HTTP error and not an MCP `isError` result. The Workhorse stream deadline (30 s) is equal to the Orbit ClickHouse limit (30 s), so Workhorse often stops the stream before Orbit can return its own clean error. ### Other triggers Any error from the go-sdk `CallTool` causes the same result. This includes: - The Orbit query stream deadline (Rails `ORBIT_DEFAULT_TIMEOUT_SECONDS = 30`). Seen in the sessions above. - The Rails Orbit gRPC deadline for commands other than queries (10 s, then `ConnectionError`, then JSON-RPC `internal_error`). From the code only. - Bad tool arguments (Rails `ArgumentError`, then HTTP 400 JSON-RPC `invalid_params`). From the code only. - Any 5xx or JSON-RPC error from the GitLab MCP server or the AI Catalog MCP servers. From the code only. Why the Duo CLI does not show this: the CLI session ran Orbit through `glab orbit remote query` in `run_command`. The 504 came back as shell output, and the LLM made the query narrower. That path does not use the Workhorse MCP client. ## Proposal In Workhorse (one MR): 1. **`duoworkflow/runner.go`:** when `CallTool` fails, send an error `ActionResponse` to DWS and keep the session open. The LLM gets the error text and can retry. 2. **`orbit/sendquery.go`:** when `McpID` is set, send deadline, stream and redaction errors as an MCP `isError` result with a clear message. REST callers keep the current HTTP status codes. Follow-up in Orbit: - Set the ClickHouse `max_execution_time` below the Workhorse stream deadline (30 s), so Orbit returns a clean `execution_error` first. That error already reaches the LLM as an `isError` result. Make the timeout error text tell the LLM how to narrow the query. ## How to find affected sessions - DWS (`gitlab-runway-production`): `resource.labels.service_name="duo-workflow-svc" jsonPayload.event="Finished ExecuteWorkflow RPC" jsonPayload.workflow_last_gitlab_status="start" jsonPayload.workflow_stop_reason="DWS_SERVER_SHUTDOWN"` - Orbit (`gl-orbit-prd`, container `gkg-webserver`): `jsonPayload.span.ai_session_id="<workflow id>"`