Add detached mode to the duoworkflow execute endpoint

What does this MR do and why?

A server-side Duo Agent Platform turn (Slack, note replies) holds a Sidekiq thread for its whole run, only to read the ndjson stream from POST /api/v4/ai/duo_workflows/workflows/:id/execute. This MR adds a detached mode to that endpoint, the Workhorse part of #634394.

When the pre-authorization response sets Detached: true, Workhorse answers 202 Accepted and keeps running the workflow after the request ends. Rails decides, so a caller can't opt in on its own. Nothing sends Detached until the Rails side lands behind a feature flag in !264231 (merged), so this MR changes no behaviour on its own. A Workhorse without this change ignores Detached and streams; the Rails MR handles both, and the flag is only to be enabled once every Workhorse instance has this change.

When the 202 is sent. A successful send of the start request does not mean Duo Workflow Service accepted it: quota and invalid-request rejections arrive on the receive side. So the handler waits for the first action, or 10 seconds after the start request was sent without one. A run that ends before then is reported with the same status codes as the streaming mode (400, 403, 409, 500), plus 503 in three cases:

• Redis is unavailable, so the workflow lock can't be taken.
• Workhorse or Duo Workflow Service drained before the run got going.
• Workhorse is already draining, so the run is not started.

The run after the 202.

• It runs on a copy of the request whose context keeps its values (correlation ID) but is not canceled when the request ends. It stays in the handler's set of runners, so a graceful shutdown still stops it and releases the lock.
• Drain. A Workhorse drain ends the run even when Duo Workflow Service doesn't acknowledge the stop request, as closing the connection does for a WebSocket session, so the run never outlives its released lock.
• New runs during a drain. A drain only stops the runs that exist when it starts, and a detached run outlives its request, so a run started during graceful_shutdown_delay would have been cut off at process exit with its lock held until it expired. Once a drain has started, detached requests get 503 before the lock is taken or Duo Workflow Service is contacted.
• Lock. On a transient Redis error a detached run fails instead of running without the lock: once the dispatching job has its 202, the lock is the only thing that keeps a retried or repeated dispatch from starting a second run. When locking is not configured, it runs without one, as other runs do (see below).
• Run limit. After 55 minutes the run sends a StopWorkflowRequest with the new reason WORKHORSE_RUN_LIMIT through the existing stop handshake. That is below the one-hour cap on a Duo Workflow Service stream and the two-hour lifetime of the user token and the lock.
• Metrics and logs. transport="detached" on connections_open and session_duration_seconds, and a run_limit teardown reason. Nobody reads the outcome of an accepted run, so a failure is logged as duo workflow: detached run failed with transport, workflow_id and error_type.

Known limitations, tracked as follow-ups on the work item:

• Duo Workflow Service has to list WORKHORSE_RUN_LIMIT in INFRA_STOP_REASONS. Until it does, it treats the limit as a user stop and marks the workflow stopped. Needed before the flag is turned on.
• No segment-end callback or resume yet. A run that ends while the workflow is still running (deploy, stream error, run limit) is caught by the stuck-workflow cleanup within about 30 minutes, which delivers a failure.
• A drain reaches detached runs only with a non-zero graceful_shutdown_delay, the same as WebSocket sessions today.
• ServerSideTurnWorker doesn't retry on 503 yet, so a dispatch that reaches a draining Workhorse shows up as a failed turn.

Why a detached run goes ahead without a lock when locking is off

Locking is off when LockConcurrentFlow is false or Workhorse has no Redis, and Rails decides both. Every other run on such an instance already goes without the single-run guarantee, so a detached run is no worse off: a retried dispatch could start a second run, just as a retried WebSocket or ndjson connection can. Rejecting detached runs there would fail every server-side turn on those setups. If Rails needs the guarantee, it can stop sending Detached when it doesn't send LockConcurrentFlow; that check belongs in pre-authorization, not in Workhorse.

Implementation notes

• detachedTransport implements clientTransport like ndjsonTransport, except that it drops checkpoints (nobody reads them), has no keepalive, and owns the run-limit timer. When the limit is reached, ReadClientEvent returns an error that ReadError maps to WORKHORSE_RUN_LIMIT, so the existing stop handshake runs unchanged.
• The runner gets two optional hooks, onStartSent and onFirstAction, both nil-safe, so existing callers and tests are unaffected.
• The run goroutine reports its outcome from the runner's error callback, before Close, so a slow close can't turn a failure into a 202. It also recovers panics on its own goroutine, as net/http does for the streaming path; a panic on one of the runner's own goroutines still takes the process down, on every path.
• classifyHTTPOutcome now takes an interface, so both transports share the status mapping.
• On a Workhorse drain, detachedTransport.SendGoingAway closes the transport, which ends Execute. A drain by Duo Workflow Service only records it, because that run ends on its own.

References

• Work item: #634394
• Audit and plan: https://gitlab.com/gitlab-org/gitlab/-/work_items/634419
• Builds on the run metrics from !263068 (merged) (merged)
• Rails side: !264231 (merged)

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

Check out !264231 (merged) for Rails, with Workhorse built from this branch.

1. In the Rails console, enable the flag for your user:

   Feature.enable(:duo_workflow_detached_server_side_execution, User.find_by_username('root'))

2. Start a server-side turn, for example by mentioning @GitLabDuo in a note on a project with Duo Agent Platform enabled.
3. In the Workhorse log, look for duo workflow: detached run accepted. The ServerSideTurnWorker job finishes within seconds.
4. The answer is still posted, by CallbackWorker, once the workflow reaches input_required or finished.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.