<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/engineering/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=634394) </details> <!--IssueSummary end--> ## Problem Server-side (environment-less) Duo Agent Platform turns, used by the Slack and note-based messaging surfaces, run like this: ``` Sidekiq ServerSideTurnWorker ──HTTP ndjson stream──► Workhorse ──gRPC ExecuteWorkflow──► DWS ``` The Sidekiq job does no work. It opens `POST /api/v4/ai/duo_workflows/workflows/:id/execute`, reads keepalives and checkpoint lines until the stream ends, then polls the workflow status and delivers the result. For the whole turn it holds a Sidekiq thread. Consequences: - **Sidekiq threads are pinned on idle I/O.** The worker has a concurrency limit of 50 and `urgency :low`; every running turn occupies one slot. Throughput is capped by the number of Sidekiq threads we are willing to park, not by Workhorse or DWS capacity. - **Sidekiq is built for short jobs.** Deploys, memory-killer restarts and job retries interrupt long jobs. When the Sidekiq side of the connection drops, Workhorse cancels the DWS stream, because the run is bound to the incoming request context. - **One connection can't outlive its ceilings.** The DWS stream is capped at one hour by its hosting platform, the user token expires after two hours and is never refreshed, and the Workhorse Redis lock has a two-hour TTL that isn't renewed. A turn that approaches any of these ends abruptly, and the only recovery is the stuck-workflow cleanup (30 minutes of inactivity). - **Nothing can reach a running turn.** On the ndjson path the client never sends events, so a stop requested in the UI doesn't reach Workhorse; the run continues until DWS notices the status change or the stream ends. ## Proposal Let Workhorse run a server-side turn **detached** from the request that started it, with Rails continuing to own the workflow. - `POST .../execute` in detached mode does what it does today up to and including opening the DWS stream and sending the start request. It then returns **202 Accepted** and continues the run on a context that is no longer tied to the request. - A run is bounded by a **run limit** (55 minutes; below the DWS one-hour stream cap and the two-hour token and lock TTLs). At the limit Workhorse stops the stream with an infrastructure stop reason, the kind DWS treats as "the client went away": the workflow status stays `running` and the next start resumes from the latest checkpoint. It then releases the lock. - Delivery to the surface is driven by the existing EventStore transition events and the `CallbackWorker`, not by the dispatching job. - Later iterations turn the run limit into **segments**: Workhorse reports each segment end to Rails, and Rails dispatches the next segment while the workflow is still `running`. This is the same lifecycle DWS and Workhorse already implement for a WebSocket client that disconnects and reconnects, with Rails playing the role of the reconnecting client. ## Scope of the first iteration Agreed while planning the first MR, to keep it small: - **In:** the detached entry point (`202`), the 55-minute run limit, the workflow lock required on transient Redis errors, detached metrics labels. - **Out, as follow-ups (see [Follow-ups](#follow-ups)):** segment-end callback and resume, watchdog, stop signal, per-process cap on detached runs, lock renewal. Whether to build the segment-end callback and the watchdog at all is not decided yet. - **Consequence:** a run that ends while the workflow is still `running` (Workhorse or DWS deploy, stream error, run limit) is not resumed or reported right away. The stuck-workflow cleanup drops it within about 30 minutes; `drop` publishes `WorkflowFailedEvent`, so `CallbackWorker` still delivers a failure. Today the blocking worker delivers that failure immediately. This is acceptable while the flag is limited to dogfooding; the callback should land before the flag goes further. ### Decisions - **How detached mode is selected.** The worker sends `?detached=true`. Rails pre-authorization reads it (Workhorse forwards the query string, not the body) and returns `Detached: true` only when the feature flag is enabled for the actor. Any client type may ask, so the browser can use it later. Workhorse reads only the pre-authorization field, never the query parameter. Reading it directly would bypass the flag, and would let any `ai_workflows` token start fire-and-forget runs that no longer cost the caller an open connection. Having the caller ask also avoids a skew bug: if the worker and pre-authorization checked the flag separately, a worker expecting a stream could get a `202` with an empty body, see `running`, and report a failure for a turn that is still going. - **Lock.** Once the dispatching job has its `202`, Sidekiq deduplication (`until_executed`) no longer spans the run. The lock is then the only thing that keeps a retried job, or a new turn for the same workflow, from starting a second run that would duplicate tool calls and checkpoints. So a detached run fails with `503` on a transient Redis error instead of running without the lock. When locking is not configured at all, it runs without one, as other runs do. - **Run-limit stop reason.** A new `WORKHORSE_RUN_LIMIT` reason rather than reusing `WORKHORSE_SERVER_SHUTDOWN`, with a DWS change to follow (see below). - **When the `202` is sent.** A successful send of the start request doesn't mean DWS accepted it: quota and invalid-request rejections only arrive on `Recv`. Workhorse waits for the first action, or 10 seconds after the start request was sent without one. A run that ends before that is reported as today (`400`/`403`/`409`/`500`), plus `503` for an unavailable lock or a drain before the run got going. ## Why this is not dangerous at this point - **Workhorse already does the long-running part.** On the WebSocket paths Workhorse holds a DWS stream, an MCP client and a Redis lock in a goroutine for as long as a web chat or IDE session lasts, often hours. Detached mode changes only who cancels the run (a timer instead of a client disconnect). It removes a Sidekiq thread that was merely waiting; it adds no new kind of work to Workhorse. - **Resume is an existing, exercised path.** DWS classifies the next start of a non-finished workflow as a retry from the latest checkpoint, and it already treats `WORKHORSE_SERVER_SHUTDOWN` and similar reasons as resumable. Every Workhorse drain and every WebSocket drop goes through this today. The run limit is the same event on a timer. - **Everything is bounded.** Run limit < DWS stream cap < token and lock TTLs, so no run can run into a ceiling. The lock guarantees at most one run per workflow at a time. - **Rails remains the source of truth and the owner.** Workflow status, checkpoints and the mailbox are unchanged. No state lives in Workhorse beyond the run in flight, which is the same as a WebSocket session today. - **No new trust boundary.** The same user token with the same `ai_workflows` scope, the same endpoint allowlist, the same action handlers, the same MCP trust rules. The server-side path already answers file and command actions with an error, so no new capability is exposed to the model. Rails, not the caller, decides whether a run is detached. - **No noisy-neighbour risk.** AI traffic is served by a dedicated `ai-assisted` fleet, so detached runs don't share pods with git or web traffic. Deploys of that fleet cut running sessions, which resume from their checkpoint, exactly as web chat sessions do on that fleet today. - **Narrow scope and a fallback.** Only the `workhorse` runtime of the run service (messaging surfaces) is affected. IDE, CLI, CI and web chat are untouched. The current blocking mode remains behind the same flag until detached mode is proven. ## Implementation plan ### Workhorse | # | Scope | Status | |---|---|---| | W1 | Decouple the run's lifecycle from the request: run on a context derived without cancellation from the request; keep registration in the handler's active-runner set so drain still reaches it. Add the detached entry point to the `execute` handler that returns `202` once DWS has accepted the start request (or an error status before that, as today). | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264218 | | W2 | Run limit: a timer that triggers the existing stop path with a new infrastructure reason (`WORKHORSE_RUN_LIMIT`) and releases the lock. Make the lock required on transient Redis errors in detached mode. | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264218. The per-process cap (`503` when full) is a follow-up. | | W3 | Segment-end callback: a Workhorse-authenticated `POST` to a Rails internal endpoint with `{workflow_id, reason, segment_duration}` on every segment end (limit, drain, DWS unavailable, stream error, stop). Best effort with a short retry. | Follow-up, **not yet decided**: don't implement until it is (see [Follow-ups](#follow-ups)) | | W4 | Stop signal: check a Redis key (`workhorse:duoworkflow:stop:<id>`) before each action and on a slow ticker; when set, send the user-level stop reason so DWS marks the workflow stopped. | Follow-up | | W5 | Lock: renew while the run is going (or set the TTL to the run limit plus grace) so the two-hour TTL is no longer load-bearing. | Follow-up | | W6 | Metrics and logs. Builds on the run metrics from https://gitlab.com/gitlab-org/gitlab/-/merge_requests/263068. | `transport="detached"` on `connections_open` and `session_duration_seconds` and the `run_limit` teardown reason are in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264218. Segment end reasons, callback failures, cap rejections and stop latency come with W2-W4. | ### Rails | # | Scope | Status | |---|---|---| | R1 | Pre-authorization for `execute` returns `Detached: true` when the caller sends `detached=true` and the feature flag is enabled for the actor. Later also `SegmentLimit`. | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264231. `SegmentLimit` is a follow-up. | | R2 | Internal endpoint for segment end (Workhorse-authenticated). Workflow terminal or in an interrupt state → nothing (transition events already delivered); `running` and reason is resumable → enqueue the next segment; non-resumable error → `drop` with a reason. Idempotent per `(workflow, segment)`. A first version could just `drop` a still-running workflow, so the failure is delivered right away as today. It must allow for DWS writing the status after the stream closes (the worker polls for this today). | Follow-up, **not yet decided**: don't implement until it is (see [Follow-ups](#follow-ups)) | | R3 | `ServerSideTurnWorker` becomes dispatch-only: mint the token, `POST execute?detached=true`, handle `202`/`400`/`403`/`409`/`503`, exit. | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264231. Errors before the `202` are reported as before; retrying `409`/`503` is a follow-up. | | R3b | Watchdog job at run limit + grace that re-dispatches if the workflow is still `running` and the lock is free, and reschedules if the lock is held. | Follow-up, **not yet decided**: don't implement until it is (see [Follow-ups](#follow-ups)) | | R4 | Delivery: `CallbackWorker` is already subscribed to `WorkflowInputRequiredEvent`, `WorkflowFinishedEvent`, `WorkflowFailedEvent`, `WorkflowStoppedEvent` and `WorkflowApprovalRequiredEvent`. Confirm that the event path posts the answer for `workhorse`-runtime turns (it calls `adapter.on_input_required`, while the worker calls `deliver_success`), then remove the inline delivery. | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264231: it did not (see below), so `CallbackWorker` now delivers on `input_required` for detached turns. The inline delivery stays for the blocking mode. | | R5 | Stop: when a `stop` is requested for a `workhorse`-runtime workflow, write the mailbox event as today and set the Redis stop key for Workhorse. | Follow-up | | R6 | Feature flag (actor: user, then namespace), admin-visible metrics on the instance, documentation. | `wip` flag `duo_workflow_detached_server_side_execution` (actor: user) in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264231. Metrics and documentation are follow-ups. | ## Discovered during implementation - **DWS only treats allowlisted stop reasons as infrastructure stops.** `INFRA_STOP_REASONS` in `duo_workflow_service/workflows/type_definitions.py` lists `WORKHORSE_SERVER_SHUTDOWN`, `WORKHORSE_WEBSOCKET_PING_FAILED` and `WORKHORSE_WEBSOCKET_PONG_TIMEOUT`. Any other reason counts as a user stop, which marks the workflow `stopped` and publishes `WorkflowStoppedEvent`. DWS replies `UNAVAILABLE`, the acknowledgement Workhorse's stop handshake waits for, only for reasons in `SERVER_SHUTDOWN_STOP_REASONS`. - **A drain reaches long-running sessions only with a graceful shutdown delay, and that is intentional.** `Handler.Shutdown` is only called through `UpgradedConnsManager`, which runs only when the health check server is configured with `graceful_shutdown_delay > 0`. Its budget is the delay minus 10 seconds, so the default `10s` stops runs at once (the stop request is still sent and the lock released). `srv.Shutdown` doesn't wait for detached runs because no handler is in flight. Without that configuration, the process exits mid-run without a stop request and the lock stays held for two hours. WebSocket sessions behave the same way, and detached runs follow them on purpose: draining long-running sessions is what `graceful_shutdown_delay` is for, so `gracefulShutdown` is not changed to stop them within `ShutdownTimeout`. - **The `ServerSideTurnWorker` comment is out of date.** It says a turn ending at `:input_required` fires no event, but `require_input` publishes `WorkflowInputRequiredEvent` for messaging sessions and `CallbackWorker` handles it. - **The stuck-workflow backstop delivers.** `CleanStuckWorkflowsService` drops `running` workflows after 30 minutes of inactivity, and `drop` publishes `WorkflowFailedEvent`, which `CallbackWorker` delivers as a failure. - **No adapter delivers on `input_required`.** `CallbackWorker` calls `adapter.on_input_required`, which is a no-op in every adapter; only `ServerSideTurnWorker` delivered the answer of a turn that pauses for the user. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264231 makes `CallbackWorker` deliver it for detached turns, marked by a `detached_turn` key in the callback context that `per_turn_reset` clears. Delivery also requires the workflow to still be `input_required`, so a late event can't post over the next turn. - **The dispatching job can't be dropped.** Some callers of `ExecuteRunService` run in web requests (Slack approval buttons, which must answer Slack within 3 seconds; MCP agent-session start; tool-approval resolution). Calling Workhorse from there would make Puma call itself through Workhorse and wait up to 10 seconds. The job stays, and now finishes in seconds. - **A Send succeeding is not acceptance.** Quota and invalid-argument rejections arrive on `Recv`, so Workhorse waits for the first action before answering `202`. ## Follow-ups Blocking the flag rollout: 1. **DWS:** add `WORKHORSE_RUN_LIMIT` to `INFRA_STOP_REASONS` and `SERVER_SHUTDOWN_STOP_REASONS`, so the run limit leaves the workflow `running` and is acknowledged with `UNAVAILABLE`. Until then a run that reaches the limit is marked `stopped`, and Workhorse's stop handshake waits for its timeout. 1. **Rails (R1, R3, R4, R6):** https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264231. 1. **Deployment order:** the Workhorse MR and the Rails MR deployed to every node before the flag is enabled; an older `CallbackWorker` doesn't deliver detached answers. Before the flag goes beyond dogfooding: 1. **Retry before acceptance:** a `409` or `503` before the `202` is reported to the user as a failure. A `503` (lock unavailable, drain) should be retried, and a `409` from a re-pushed job whose first run is still going should not post a failure and claim the delivery. 1. **Concurrency limits:** a per-process cap on detached runs (`503` when full), and possibly a per-user cap on detached runs in flight. Any client type may request detached mode, and without a held connection the rate limit, the lock and the per-process cap are the only bounds on how many runs one user can start. Not yet decided, so not to be implemented until a decision is recorded here: 1. **Segment-end callback (W3, R2)** would report or resume a run that ends while the workflow is still `running`, right away instead of after the 30-minute cleanup. 1. **Watchdog (R3b)** would re-dispatch a run whose segment-end callback was lost. Whether these are worth their cost depends on how often detached runs end while still `running` during dogfooding (deploys, stream errors, the run limit) and how much the 30-minute wait costs users. Until then, the stuck-workflow cleanup is the backstop. Later: 1. Stop signal (W4, R5), lock renewal (W5), run limit supplied by Rails (`SegmentLimit`), and the remaining detached metrics (W6). ### Rollout 1. Dogfood on the gitlab-org Slack integration with the flag on for a small group; compare turn success rate, repeated-step rate at run-limit boundaries, and Sidekiq thread occupancy against the blocking mode. 2. Enable for GitLab.com messaging surfaces. 3. Default on; keep the blocking mode for one release as a fallback, then remove it. ## Exit criteria - No Sidekiq thread is held for the duration of a server-side turn; the dispatching job completes in seconds. - Turns survive Workhorse and DWS deploys and the one-hour DWS stream cap without user-visible failure (a boundary costs one resume). Needs the segment-end callback, if it is decided on. - A stop requested in the UI reaches a running server-side turn within seconds. Needs the stop signal follow-up. - A lost callback is recovered by the watchdog, and a lost watchdog by the stuck-workflow cleanup, in chaos tests (killed Workhorse pod, Redis failover, DWS drain). Applies only if the callback and watchdog are decided on. - Flag off restores today's behaviour.