Retry refused server-side Duo turns and stop resending on re-runs

What does this MR do and why?

Implements the "Retry before acceptance" follow-up of detached server-side execution.

When Workhorse answers execute with 409 (workflow lock held) or 503 (draining, or the lock store unavailable), nothing has started. ServerSideTurnWorker still reported both to the user as failures. For a 409 that is worse than a wrong message: posting the failure claims the delivery, which suppresses the answer of the run that holds the lock.

Now the worker:

  • Retries 409 and 503 through Sidekiq on a short schedule (10, 20, 30 seconds plus jitter), since a user is waiting. Once the retries run out, it reports the failure as before, except for a 409 while the workflow is running, whose outcome belongs to the running turn.
  • Skips a 409 straight away when the turn is detached and the workflow is running. That turn delivers its own outcome; it is usually this job's own run from before Sidekiq re-pushed it.
  • Doesn't run a turn again when a retry finds it already answered. Without this, a re-pushed job could retry into a free lock after its first run finished and repeat the turn. ExecuteRunService clears the claim before it enqueues a new turn, so first attempts are unaffected.

The 503 retry also applies to the blocking mode: nothing started in either mode, so starting again is safe.

Sessions with no messaging surface, such as a catalog flow's or an MCP session's, get the same retries. Once the retries run out, the worker drops the session as it does for any other failed start. The exhausted hook also releases the turn claim, which a raise keeps for the retry, so an MCP user isn't refused for the claim's full hour.

It also stops a re-run from resending the turn's message. ai-gateway reads a goal or approval sent at a wait as the user's next reply, so when Sidekiq ran the job again after the run took the message (a delivery retry, a retry after another error, a deploy re-push), the agent got the same message twice. The worker now records a per-turn turn_started marker in the callback context before it hands the message to the run, and ExecuteRunService clears it with the other per-turn keys. When a re-run finds the marker:

  • If the turn ended at a wait or finished, it doesn't call Workhorse again and only retries the delivery. A start without a goal is rejected at a wait for input.
  • If the run is still going, it resumes without a goal or approval, which continues from the latest checkpoint.

A 409 or 503 clears the marker only when this attempt set it, so a retry hands the message over only when no run took it.

Trade-off: a concurrent message during a running detached turn

Two messages can both pass RunEvent#valid_for? while the workflow is input_required and enqueue two turns. The second gets a 409 while the first is running and is now left unanswered. Before, it got "I'm still working on an earlier message", but that also claimed the delivery and dropped the first turn's answer. Telling this case apart from a re-pushed job would need the worker to record which job started the run. That doesn't seem worth it for a case this rare.

The same race can also clear the first turn's turn_started marker: the second message's enqueue resets the per-turn keys, and its job clears the marker after its 409. If Sidekiq then re-pushes the first job, the rerun resends the first message. Allowing one turn in flight per workflow would fix both, for example with the TurnClaim the MCP resume route uses. That is a follow-up.

Production context

From GitLab.com Workhorse logs, 2026-10-22 to 2026-10-29: 51 detached and 331 streaming execute requests, with no 409 or 503. So this hardens the path before the flag goes beyond dogfooding; it doesn't fix an active problem.

References

Closes #632217

Screenshots or screen recordings

Not applicable: no UI changes.

How to set up and validate locally

  1. Enable the flag for your user in the Rails console:

    Feature.enable(:duo_workflow_detached_server_side_execution, User.find_by_username('root'))
  2. Start a server-side turn (for example, mention Duo in Slack) while a detached run already holds the workflow's lock, or while Workhorse is draining.

  3. Check log/sidekiq.log: the ServerSideTurnWorker job is retried instead of posting a failure. With a detached turn running, the job finishes after logging Duo Messaging: Workflow locked by a running detached turn.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Igor Drozdov

Merge request reports

Loading
Loading