## Why Every Duo flow that starts from a background job — a Slack mention, an `@GitLabDuo` comment, a webhook — runs today as a CI job, because CI is the only place the platform can execute a flow without a browser or IDE attached. For flows that need a repository that is right. For API-only flows it is the reason they are slow, need a project, and run under composite identity bound to one top-level group. Workhorse already knows how to run a flow: it holds the gRPC stream to the Duo Workflow Service (DWS) and executes the tool calls for agentic web chat. What is missing is a way for a server-side caller to start that without a CI job in between. This issue adds it. ## What this issue delivers An HTTP endpoint, `POST /api/v4/ai/duo_workflows/workflows/:id/execute`, that lets a server-side caller (Rails, from a Sidekiq job) start a flow directly in Workhorse and read the run back as a single chunked ndjson stream of checkpoints. Workhorse answers, on the caller's behalf, the tool calls that don't need a client (`RunHTTPRequest`, `RunMCPTool`, ...) for the run's lifetime. No CI job, no project, no service account — the run acts as the user, in any top-level group. **Scope correction:** the caller still holds the connection open for as long as the run takes, the same as the existing WebSocket path — it does not get an immediate acknowledgement and walk away. Making the call fully non-blocking would need either Workhorse or DWS to support a detached execution mode, and neither was built for that; it was discussed and deliberately left out of this effort as a separate, larger initiative for later, rather than a hard blocker (we already have precedent for jobs running longer than five minutes, so the current approach is not wasteful even if a detached mode arrives later). Two proofs of concept existed before this issue; this issue productionises the shape they proved out (Workhorse starting and driving the run itself), not the detach that neither of them had either: - [!246709](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/246709) (Igor) — `POST /ai/duo_workflows/direct`. Carries governance, MCP config and model metadata; streams every DWS action back. - [!234471](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/234471) (Kinshuk) — a `senddata` injecter; reads the answer from the database afterwards. Take Igor's payload and Kinshuk's stance of not reading the stream. Delivered in: - [!254410](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/254410) — introduces a `clientTransport` interface in the Workhorse duoworkflow runner, so it can be driven by something other than a WebSocket. - [!254443](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/254443) — the Rails pre-authorization endpoint for `:execute`, sharing the WebSocket path's connection-config logic (`Ai::DuoWorkflows::ConnectionConfigService`). - [!254503](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/254503) — the Workhorse handler for `:execute`: opens the DWS stream itself, streams checkpoints back, answers other actions automatically, and reuses the existing per-workflow lock so a second concurrent run gets a clean `409`. ## Done when - ~~Rails can start a `slack_assistant` run for a session with a user token and gets a `202` within a second; the Slack event worker finishes without waiting for the run.~~ Descoped (see "Scope correction" above): the caller still holds the connection for the run's duration. What's delivered instead is a single chunked HTTP response that streams the run, so the caller no longer needs a CI job to do so. - The run completes, the session reaches `finished`, and the existing `CallbackWorker` posts the answer to Slack with no CI job involved. *(Depends on the Slack-side wiring, tracked in [#628544](https://gitlab.com/gitlab-org/gitlab/-/work_items/628544), which this issue blocks.)* - The agent acts in a top-level group other than the user's default namespace when asked. — delivered: authorization resolves a composite-identity token to its linked human user before the workflow lookup, same as the WebSocket path. - Killing workhorse mid-run leaves the session in a terminal state within a short, defined time — not in `running`. — partially delivered: a graceful Workhorse shutdown still sends the stop request; a caller hangup is treated as a stream cancellation, same as a WebSocket client disconnecting. - Starting a second run for an active session fails with a clear error. — delivered: reuses `workflowLockManager`; a concurrent call gets `409`. - The pre-authorisation code is shared with the WebSocket action, not copied. — delivered via `Ai::DuoWorkflows::ConnectionConfigService`. <details> <summary><strong>Requirements and proposal</strong> — for whoever implements this</summary> Written as requirements from the messaging side. The shape of the implementation is the implementer's call. ### Requirements 1. **Rails starts a run and returns.** One call carrying the session id, the flow identity (`flow_config_id`, schema version, flow version), the goal, and the token. Workhorse acknowledges (`202` plus ids) and continues on its own. The Sidekiq job that received the Slack event must not wait. Igor's POC blocks the caller for up to ten minutes, Kinshuk's for five — a Sidekiq thread per Slack conversation, and a run that dies on every deploy. 2. **Rails needs nothing from the stream.** DWS writes its own state back over the same gRPC connection, and workhorse serves those calls: checkpoints to `POST /ai/duo_workflows/workflows/:id/checkpoints`, status to `PATCH /ai/duo_workflows/workflows/:id`. The messaging adapter already renders progress from checkpoints and posts the final answer on `WorkflowFinishedEvent`. Workhorse must stay alive as the tool executor for the run's lifetime; nothing has to read what it streams. 3. **Runs as the user.** A plain user OAuth token from `CreateOauthAccessTokenService` (scopes `ai_workflows` + `mcp`), not a composite-identity token. Same as agentic web chat. This is what lets the agent act across top-level groups. 4. **Failure is reported.** Today Slack learns that a run died from a CI event. Without CI a crashed run leaves the thread spinning forever. If the stream dies, DWS errors, or the run times out, the session must reach a terminal state so the lifecycle-events issue can tell the surface. Whether workhorse calls Rails or Rails watches for silence is the open design question; the existing 30-minute `CleanStuckWorkflowsService` is far too slow for a chat surface. 5. **One active run per session.** Workhorse's `workflowLockManager` (Redis, per workflow) already does this for the WebSocket path. Take it. A second start while one is active must fail cleanly. 6. **Resume.** The same call carrying an `approval` decision (approve / reject / reject with message) instead of a goal resumes a paused session. DWS supports this statelessly today (`StartWorkflowRequest.approval` plus the session's checkpoints). Slack's approve/deny buttons will call this. Can be a follow-up MR; the contract should allow for it from the start. 7. **Cancellation and shutdown.** A way to stop a run, and graceful behaviour on workhorse restart: at minimum, a run killed by a deploy ends in a terminal state rather than silence. A registry of active runs (`workflow_id → stream`) is what both the lease and cancellation hang off. 8. **Parity with the WebSocket path.** Governance policies, MCP config, model selection, self-hosted DWS and prompt-cache settings travel through the pre-authorisation call today (`ee/lib/api/ai/duo_workflows/workflows.rb`, `:ws`). Igor's POC clones that action; share it instead. 9. **GitLab MCP tools for the flow.** `McpConfigService#gitlab_mcp_enabled?` hands the GitLab MCP server to a session only for `chat` / `agentic_chat/v1` or catalog agents with tools. Add `slack_assistant/v1` (a registry attribute would be cleaner than extending the constant). With the flow's `environment: chat`, DWS adds the tools automatically. Behind the existing `mcp_client` flag. ### Details - Namespace-level sessions are a separate issue, but this endpoint must accept a namespace as the container, not only a project. `CreateWorkflowService` already does; `StartWorkflowService` (CI) is the one that refuses. - The v1 flow's `toolset:` decides what the agent sees; the session's privileges cap it. On this runtime there is no executor for file or shell tools, so over-granting is inert — the registration issue narrows privileges anyway. - Limits: request body size, per-run timeout, concurrent runs per instance. The POC has none. - Metrics and logs: runs started, finished, failed, duration; workflow id on every log line. - Latency benchmark against the CI path for the same flow, so the improvement is a number. - This runtime plugs into `Runs::StartService` as `Runtimes::Headless` (runtime seam issue). ### Open questions for the implementer - Failure reporting: does workhorse tell Rails, or does Rails watch? See requirement 4. - Is a detached goroutine in workhorse acceptable operationally (drain on deploy, limits), or does this want a small separate service? The POCs assume the former. **Resolved for this issue: neither Workhorse nor DWS supports detached execution today, and building it was judged a separate, larger initiative — see "Scope correction" above.** - Igor may already track parts of this elsewhere; if so, link or close this as a duplicate — the requirements above are what the Slack side needs either way. </details> ## Not in this issue - Choosing this runtime from the messaging adapter — runtime seam issue. - Publishing the failure/approval events the adapter listens to — lifecycle events issue. - Namespace-level sessions in Rails — separate issue. - Slack approve/deny UI — approvals issue. - A fully non-blocking / detached execution mode (the HTTP caller still holds the connection for the run's duration) — deferred to a separate, larger initiative in Workhorse or DWS. ## Depends on / unblocks - Depends on: nothing to start. For the end-to-end check: the `slack_assistant` flow and its registration. - Unblocks: namespace-level sessions, resume from Slack approvals, the cutover.