Run Duo Workflow actions concurrently in Workhorse

Igor Drozdovrequested to merge
634393-duoworkflow-parallel-actions into master

What does this MR do and why?

Workhorse now runs the Duo Workflow actions it executes itself (RunHTTPRequest, RunMCPTool for tools it knows, TrackLlmCallForSelfHosted) concurrently, on a bounded pool per run. This is the worker pool from #634393.

Until now, Workhorse ran actions one at a time. A step with five parallel tool calls took the sum of their durations, and checkpoint writes waited behind slow MCP calls. Duo Workflow Service already matches responses by request ID, so the order in which they complete doesn't matter.

No behaviour change until Rails opts in. The pool size comes from a new MaxConcurrentActions field in the pre-authorization response. Rails doesn't send it yet, and at 0 or 1 actions run on the receiving goroutine exactly as before. The Rails field and its feature flag come in a follow-up MR.

How it works:

  • Ordering: actions forwarded to the client (checkpoints, file and command actions, MCP tools from servers configured in the IDE) are still written by the receiving goroutine, in the order they arrive.
  • Back-pressure: when every worker is busy, the receive loop waits for one to free up before it reads the next action, as the serial loop does today with a limit of 1.
  • Errors: a failed action is still answered with an error response. A response that can't be sent still ends the run, and it now also cancels the other actions in flight.
  • Panics: a worker recovers a panic, logs it with the stack, and answers the action with a generic error. Without this, a panic would crash Workhorse, because workers run outside net/http's recovery.
  • Close: actions still in flight are canceled and waited for, up to the stop timeout, before the stream is closed. This way no response is sent on a closed stream.

Concurrency safety:

  • Self-hosted tracking: it sends on the cloud tracking stream and then receives the acknowledgment. A mutex now keeps each send paired with its receive, so concurrent calls don't take each other's acknowledgments.
  • MCP: the go-sdk ClientSession handles concurrent CallTool on one session. A new test holds concurrent calls to the same server until all of them arrive. The shared external MCP transport already allows 10 idle connections per host.

New metrics:

Metric Labels
gitlab_workhorse_duo_workflow_action_queue_wait_seconds type (http, mcp, self_hosted_tracking)
gitlab_workhorse_duo_workflow_action_pool_saturated_total

This MR is based on master and doesn't depend on !263068 (merged), which adds the per-action metrics (actions_total, actions_in_flight, action_duration_seconds). The only overlap is the http/mcp/self_hosted_tracking type values and the classification in runner.go. Whichever lands second needs a small rebase.

Choices worth a second look
  • Serial at size 1, not a 1-worker pool. With one worker, a client-bound action could be written while an HTTP action was still running, which isn't how things work today. Running inline at size 1 or below reproduces the current behaviour exactly.
  • Close cancels before it waits. By the time Close runs, Execute has returned, so DWS won't use any response still in flight. Canceling first keeps the wait short, and the timeout only matters when an action ignores cancellation.
  • Only the first fatal error is reported. The pool has its own slot in Execute's error channel and sends to it at most once, so a worker can never block on a full channel.

References

Screenshots or screen recordings

Not applicable: no UI changes.

How to set up and validate locally

  1. Until the Rails MR lands, make Rails send the setting: in ee/app/services/ai/duo_workflows/connection_config_service.rb, add MaxConcurrentActions: 8, next to TimeoutHTTPRequests.
  2. Open Agentic Chat in the web UI and ask something that makes the agent call several tools in one step, such as "Summarise the three most recent issues and merge requests in this project".
  3. In the Workhorse logs, confirm the Executing HTTP request lines for one step appear together, before their responses, instead of alternating with them.
  4. Fetch /metrics from Workhorse's Prometheus listener and confirm gitlab_workhorse_duo_workflow_action_queue_wait_seconds is present.
  5. Remove the line from step 1 and confirm the actions run one at a time again.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports