Sign Workhorse MCP client requests with the API JWT

What does this MR do and why?

Since 2026-07-28 every Orbit MCP tool call in the Duo Agent Platform gets 403 from /api/v4/orbit/mcp and the chat workflow is torn down. The user sees the agent stuck on "finding an answer" forever. This is the regression behind the Orbit half of the DAP tool-call hangs reported on Jul 28-29.

The security hardening in 9d036170 (gitlab-org/security/gitlab!6169) requires a fresh Gitlab-Workhorse-Api-Request JWT before Rails emits any Gitlab-Workhorse-Send-Data response, and Orbit tools/call replies with senddata. Workhorse's embedded MCP client calls the internal MCP endpoints directly with a custom transport that never signed that JWT, so verify_workhorse_api! raises JWT::DecodeError, Rails renders 403, and Workhorse treats the failed CallTool as workflow-fatal and closes the WebSocket.

The fix wraps the internal MCP transport in secret.NewRoundTripper, the same signing round tripper the proxy path uses, so both internal MCP servers (gitlab, orbit) present a valid JWT. Two-line production change.

Root cause detail and why not an exemption
  • Failure chain on GDK at latest master: Rails api_json.log shows 403 on POST /api/v4/orbit/mcp with UA GitLab-Workhorse-Mcp-Client; Workhorse logs error executing workflow: ... calling "tools/call": sending "tools/call": Forbidden; the WebSocket closes and the DWS stream dies, which renders as a permanent "finding an answer" in Duo Chat.
  • On production the 403s from this UA went 0 (Jul 27) to 62 (Jul 28) to 158 (Jul 29), matching the auto-deploy rollout of 9d036170. Intermittency during the window was the canary/gprd pod mix.
  • initialize and tools/list are handled inline by Rails and were unaffected; every real Orbit tool invocation relays to GKG via senddata and failed.
  • Exempting orbit/mcp from verify_workhorse_api! instead would reopen the hole the security MR closed: the endpoint is also reachable by external MCP clients on the same route, and its senddata must stay Workhorse-only.
  • The embedded client lives inside Workhorse, which holds the Workhorse secret, so minting the JWT in its transport is the intended trust model.

References

Screenshots or screen recordings

GDK at latest master, web Duo Agentic Chat, Orbit enabled, prompt asking for an orbit_query_graph lookup.

Before (master) After (this MR)
before-hang after-success

Before: the workflow dies on the first query_graph call (Rails 403, Workhorse calling "tools/call": Forbidden) and the chat spins forever. After: the same prompt completes; the senddata relay to GKG executes and the agent reports real query results.

How to set up and validate locally

  1. GDK with the Orbit/GKG stack, DWS, and AI Gateway running; Orbit enabled for the user and duo_workflow_mcp_enabled on the project's root namespace.
  2. In web Duo Agentic Chat ask: "Use the orbit_query_graph tool to find Projects whose full_path starts with '<a real group>/'. Report the names."
  3. On master the chat hangs and log/gitlab-workhorse/current logs calling "tools/call": Forbidden while log/api_json.log logs 403 on POST /api/v4/orbit/mcp from UA GitLab-Workhorse-Mcp-Client. With this MR the tool call returns data and the chat answers.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Merge request reports

Loading
Loading