Sign Workhorse MCP client requests with the API JWT
What does this MR do and why?
Since 2026-07-28 every Orbit MCP tool call in the Duo Agent Platform gets 403 from /api/v4/orbit/mcp and the chat workflow is torn down. The user sees the agent stuck on "finding an answer" forever. This is the regression behind the Orbit half of the DAP tool-call hangs reported on Jul 28-29.
The security hardening in 9d036170 (gitlab-org/security/gitlab!6169) requires a fresh Gitlab-Workhorse-Api-Request JWT before Rails emits any Gitlab-Workhorse-Send-Data response, and Orbit tools/call replies with senddata. Workhorse's embedded MCP client calls the internal MCP endpoints directly with a custom transport that never signed that JWT, so verify_workhorse_api! raises JWT::DecodeError, Rails renders 403, and Workhorse treats the failed CallTool as workflow-fatal and closes the WebSocket.
The fix wraps the internal MCP transport in secret.NewRoundTripper, the same signing round tripper the proxy path uses, so both internal MCP servers (gitlab, orbit) present a valid JWT. Two-line production change.
Root cause detail and why not an exemption
- Failure chain on GDK at latest master: Rails
api_json.logshows 403 onPOST /api/v4/orbit/mcpwith UAGitLab-Workhorse-Mcp-Client; Workhorse logserror executing workflow: ... calling "tools/call": sending "tools/call": Forbidden; the WebSocket closes and the DWS stream dies, which renders as a permanent "finding an answer" in Duo Chat. - On production the 403s from this UA went 0 (Jul 27) to 62 (Jul 28) to 158 (Jul 29), matching the auto-deploy rollout of 9d036170. Intermittency during the window was the canary/gprd pod mix.
initializeandtools/listare handled inline by Rails and were unaffected; every real Orbit tool invocation relays to GKG via senddata and failed.- Exempting
orbit/mcpfromverify_workhorse_api!instead would reopen the hole the security MR closed: the endpoint is also reachable by external MCP clients on the same route, and its senddata must stay Workhorse-only. - The embedded client lives inside Workhorse, which holds the Workhorse secret, so minting the JWT in its transport is the intended trust model.
References
- Security hardening that surfaced this: 9d036170 (gitlab-org/security/gitlab!6169, "Implement Workhorse JWT for all senddata requests")
- Sibling DAP regression (approval loop, different defect): !247570 (merged) and https://gitlab.com/gitlab-org/gitlab/-/issues/607151
Screenshots or screen recordings
GDK at latest master, web Duo Agentic Chat, Orbit enabled, prompt asking for an orbit_query_graph lookup.
| Before (master) | After (this MR) |
|---|---|
![]() |
![]() |
Before: the workflow dies on the first query_graph call (Rails 403, Workhorse calling "tools/call": Forbidden) and the chat spins forever. After: the same prompt completes; the senddata relay to GKG executes and the agent reports real query results.
How to set up and validate locally
- GDK with the Orbit/GKG stack, DWS, and AI Gateway running; Orbit enabled for the user and
duo_workflow_mcp_enabledon the project's root namespace. - In web Duo Agentic Chat ask: "Use the orbit_query_graph tool to find Projects whose full_path starts with '<a real group>/'. Report the names."
- On master the chat hangs and
log/gitlab-workhorse/currentlogscalling "tools/call": Forbiddenwhilelog/api_json.loglogs 403 onPOST /api/v4/orbit/mcpfrom UAGitLab-Workhorse-Mcp-Client. With this MR the tool call returns data and the chat answers.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.

