Add Rails client for Duo Workflow server-side execution

Igor Drozdovrequested to merge
feature/duo-workflow-server-side-execution-service into master

What does this MR do and why?

GET /api/v4/ai/duo_workflows/ws upgrades to a WebSocket and proxies between a client and the Duo Workflow Service (DWS). In that model the client is the executor: DWS sends it Action messages, the client runs them and answers. That works when there is a client capable of running something.

Some callers cannot execute anything. A chat turn arriving from a Slack integration and handled in a Sidekiq job has no filesystem, no shell, and no way to answer a tool call. Today the only way to serve those callers is to start a CI job whose sole purpose is to hold the gRPC stream open and answer the actions that Workhorse already knows how to answer.

Two merge requests added a server-side execution endpoint for exactly this case, POST /api/v4/ai/duo_workflows/workflows/:workflow_id/execute. Workhorse opens the DWS gRPC stream itself, answers the actions, and streams them back as newline-delimited JSON over a single chunked response. This MR adds the Rails-side client for that endpoint: Ai::DuoWorkflows::ServerSideExecutionService.

Usage is Ai::DuoWorkflows::ServerSideExecutionService.new(workflow:, goal:, approval: nil).execute { ... }. The service mints an ai_workflows-scoped OAuth token for the workflow's own user through Ai::DuoWorkflows::WorkflowContextGenerationService#generate_oauth_token, sends a protojson StartWorkflowRequest as the body, and blocks until the stream ends. It is therefore only safe to call from a background job.

Chat turns run as the requesting user, the same way Web Agentic Chat does. There is no service account and therefore no composite identity, and a tool call that needs human approval pauses the flow rather than being pre-approved server-side.

Everything the Rails route reads through params has to be a query parameter. Workhorse's pre-authorization subrequest to Rails forwards the original request's method, rebased URL and headers, but never its body. The JSON body is read only by Workhorse itself, to build the StartWorkflowRequest it sends to DWS.

The stream content is deliberately not parsed. DWS persists checkpoints through the internal API as the flow runs, so the workflow row is the source of truth for both progress and outcome. Each action line is used only as a tick and handed to the caller's block, so the caller can re-read the workflow and render intermediate progress while the turn is still running. An earlier prototype parsed ui_chat_log out of the streamed checkpoints; that duplicates ProgressReader and breaks once incremental checkpoints are in play.

Decisions worth calling out

Two failure conditions are distinguished, because Workhorse commits the response header lazily. A failure status arrives before anything has been streamed, so it maps straight to a reason: 409 to :workflow_locked, 403 to :forbidden, anything else to :execute_workflow_failed. A connection lost mid-turn says nothing about whether the turn completed, so there the workflow's status decides rather than the exception. 403 maps to the generic :forbidden instead of a quota-specific reason because both Workhorse quota rejections and Rails pre-authorization denials surface as 403.

The turn outcome comes from the workflow status, not from the stream. The ndjson stream has no in-band terminal record, by design, so that its line format stays exactly what WebSocket clients already parse. The service treats the awaiting_input and completed status groups as a completed turn, and polls up to three times at one second, because the stream closing and DWS reporting the new status over the internal API are not ordered relative to each other.

read_timeout is 5 minutes and there is no cap on total turn duration, matching the WebSocket endpoint. Workhorse writes a keepalive every 20 seconds, so the read timeout only trips on a wedged connection.

Rendering progress is advisory. An exception raised by the caller's block is tracked and swallowed, so a failure to post a progress update neither abandons the turn nor gets misreported as a transport failure.

allow_local_requests: true is required because the target URL is this instance's own, built from Gitlab.config.gitlab.url plus the workflow ID. It has no user-controlled component.

Known follow-up

The minted OAuth token is not revoked when the turn ends; it expires after two hours, which is what the WebSocket path does today. Revoking it eagerly needs care, because DWS writes its final checkpoints through Workhorse using that same token, so a naive revoke races with the last checkpoint write.

Scope

This MR adds exactly two files and changes none. There is no caller yet. Follow-up merge requests add the turn service and Sidekiq worker that drive it, and wire Slack chat turns to them behind a feature flag. Splitting the client out keeps the HTTP contract with Workhorse reviewable on its own.

There is no changelog entry, because nothing calls the service yet and so there is no user-facing change; that arrives with the merge request that wires up Slack. There is no documentation change either, because the endpoint is already documented by the two merge requests linked below.

The spec has 28 examples, covering the request query, headers and body; the streaming logic (one action per line, keepalive lines skipped, an action split across fragments, several actions in one fragment, an incomplete trailing action, no block supplied, and a raising block); each pre-stream failure status; a connection lost before streaming and mid-turn; and every status the turn can legitimately end on.

References

Screenshots or screen recordings

Does not apply. There is no user-visible change.

How to set up and validate locally

The service has no caller, so validation is the spec suite and RuboCop:

bundle exec rspec ee/spec/services/ai/duo_workflows/server_side_execution_service_spec.rb
bundle exec rubocop ee/app/services/ai/duo_workflows/server_side_execution_service.rb

Both pass: 28 examples, 0 failures, and no RuboCop offenses.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports