<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/engineering/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=634393) </details> <!--IssueSummary end--> ## Problem Workhorse executes Duo Workflow actions one at a time. In `workhorse/internal/ai_assist/duoworkflow`, the agent-message loop is: ``` for { action := streamManager.Recv() handleAgentAction(ctx, action) // synchronous: HTTP to Rails, MCP call, or write to client } ``` The next `Recv` doesn't happen until the current action has fully completed. This is at odds with the other side of the stream: - **DWS issues tool calls concurrently.** The agent tool node runs each tool call as its own task and gathers them; every action carries a `requestID`, and the outbox resolves responses by that ID in any order. A step with five parallel tool calls is designed to take as long as the slowest one. Through Workhorse it takes the sum. - **Checkpoint saves queue behind tool calls.** DWS persists checkpoints by sending `RunHTTPRequest` actions to the Rails checkpoints API. Those sit in the same serial queue, so a slow MCP call or a large API response delays the checkpoint that the UI, the messaging progress worker and the resume path depend on. - **Slow external MCP servers stall the whole run.** One slow instance-configured MCP server blocks every other action, including the GitLab API calls that have nothing to do with it. This affects all three transports Workhorse serves (IDE/CLI WebSocket, web chat WebSocket, server-side ndjson), since they share the runner. ## Proposal Execute Workhorse-handled actions (`RunHTTPRequest`, `RunMCPTool`, `TrackLlmCallForSelfHosted`) concurrently through a bounded per-run worker pool, while keeping client-bound actions (`NewCheckpoint`, file and command actions forwarded to the IDE/CLI) in receive order. ### Design 1. **Receive loop stays single-threaded.** It classifies each action: - **Client-bound** (forwarded with `writeActionToClient`): written inline, in order, as today. Checkpoints and client actions keep their relative order, which the browser and the LSP executor rely on. - **Workhorse-handled** (HTTP, MCP, self-hosted tracking): submitted to the run's worker pool. The loop immediately continues with `Recv`. 2. **Bounded pool per run.** A semaphore of `N` workers (default 8, configurable; reported to Workhorse through the pre-authorization response so it can be tuned from Rails without a Workhorse release). When the pool is full the receive loop blocks, which applies back-pressure to DWS exactly as the serial loop does today, just at `N` instead of 1. 3. **Responses go through the existing serialised send.** `streamManager.Send` is already guarded by a mutex because two goroutines write to the stream today; a third source changes nothing in that contract. 4. **Per-action error isolation.** A handler error already produces an `ActionResponse` carrying the error instead of ending the run; keep that, add a `recover` per worker so a panic in one action is converted into an error response and logged rather than taking the run down. 5. **Cancellation and drain.** Workers derive from the run context, so a client disconnect, a stop request or a Workhorse drain cancels in-flight actions as today. At run end the loop waits for workers to finish (bounded by the existing stop timeout) before closing the stream, so no response is sent on a closed stream. 6. **Observability.** Add per-run metrics: actions in flight, queue wait, action duration by type and outcome, pool saturation events. These show the gain and expose a misbehaving MCP server. 7. **Rollout switch.** Enable by a field in the Rails pre-authorization response (same mechanism as `TimeoutHTTPRequests`), behind a feature flag per actor. Default off until measured; `N = 1` reproduces today's behaviour exactly. ### Out of scope - Changing the DWS contract. DWS already awaits responses by `requestID`; nothing on its side changes. - Parallelising actions that are forwarded to the IDE/CLI. Their execution happens in the client. ## Why this is safe - **The contract already permits it.** Every action has a `requestID` and DWS resolves responses by ID. Completion order is not part of the contract; the serial loop is an implementation accident, not a guarantee anybody relies on. (The legacy outbox fallback for responses without a `requestID` is irrelevant here: Workhorse always sets it.) - **Ordering that matters is preserved.** Client-bound actions and checkpoints are still written in receive order by the single receive goroutine. Only independent server-side calls run concurrently. - **The shared state is already protected.** The gRPC send path has its mutex, the WebSocket write path has its mutex, and the MCP manager guards its map with a `RWMutex`. The one item to verify during implementation is that the MCP client library tolerates concurrent `CallTool` on the same server connection; if it doesn't, use one client per in-flight call or a per-server lock, which still parallelises across servers and across HTTP actions. - **Rails already receives these calls concurrently** from the CI executor and the IDE, which don't serialise; Workhorse simply stops being the outlier. - **Bounded.** The pool caps concurrency per run; per-process limits and the existing HTTP action timeout (behind `timeout_dap_http_requests_in_workhorse`) bound the blast radius of a slow upstream. - **Reversible at runtime.** Flag off, or `N = 1`, restores today's behaviour without a deploy. - **The security rules are untouched.** Header handling, forwarded-for chains, response size limits and MCP trust rules all live in the handlers, which are called unchanged; only the scheduling around them changes. - **AI traffic runs on its own fleet** (`ai-assisted`), so higher per-run concurrency doesn't compete with git or web requests on the same pods. ## Implementation plan | # | MR | Scope | |---|---|---| | 1 | Workhorse | Extract action classification (client-bound vs Workhorse-handled) from `handleAgentAction`; no behaviour change. Add the metrics (in flight, duration by type and outcome) with the current serial execution so there's a baseline. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/263068 | | 2 | Workhorse | Add the bounded worker pool with `recover`, context propagation, and drain-before-close. Default pool size 1 (no behaviour change). Add the queue wait and pool saturation metrics: they need the pool, and with serial execution they would always read zero. Unit tests with a fake stream: ordering of client-bound writes, responses for all submitted actions, cancellation mid-flight, panic isolation. Run under the race detector. | | 3 | Workhorse | Verify MCP client concurrency; add a per-server lock or per-call client if required. Integration test with a slow fake MCP server and a fast HTTP action to show the HTTP action isn't blocked. | | 4 | Rails | Add `ParallelActions`/`MaxConcurrentActions` to the pre-authorization response behind a feature flag (actor: user), defaulting to 8 when enabled. | | 5 | Rails + Workhorse | Enable for gitlab-org dogfooding on the ndjson (server-side) path first, where there is no client to interact with, then web chat, then IDE/CLI. Compare step latency and checkpoint lag before/after using the metrics from MR 1. | | 6 | Docs | Update the AI-assisted features architecture page: actions run concurrently, bounded, ordering guarantees. | ## Exit criteria - A step with `k` parallel tool calls completes in roughly the time of the slowest call, not the sum. - Checkpoint lag (time from DWS sending a `RunHTTPRequest` for a checkpoint to the Rails write) no longer correlates with concurrent MCP call duration. - No change in ordering of client-visible events (checkpoint tests on the web chat and IDE paths). - Flag off reproduces the baseline metrics from MR 1.