Draft: Deliver each Duo turn's answer where the turn came from

What does this MR do and why?

When moving to the newly planned chat engine, Duo Developer sessions on developer/4.0.0 (the new flow we plan to run on that engine, behind the flag duo_developer_continuable_sessions) rest on the new idle status after each answer instead of finishing. With the work behind gitlab-org#23827 we are planning to allow continuing a session from the session sidebar as well.

Today an answer is posted to the thread only when a session finishes, so a resting session never posts its answer. And the callback context lives for the whole session, so a follow-up from the session page would post its answer into the original thread. For example, when Duo Developer was started by mentioning it in a comment, that comment would get a random reply from a later turn.

After this MR:

  • A mention gets its answer as soon as the session pauses, not when it finishes.
  • A thread gets the answer to the message it sent, and nothing else. Follow-ups from the session page, MCP or the Duo CLI show their answers in the session only. The thread gets no further answers, progress, approval cards or failure replies from those turns.
  • A delayed or retried delivery still posts the right answer, once, even if the next turn has already started.
  • One-shot flows are unchanged: developer/3.0.0, code review, Slack assistant, webhooks and delegated sessions deliver as before.
How it works (for reviewers) A callback context answers one turn: the one its surface's message started.
  • Turn ended event. Resting idle publishes the new WorkflowTurnEndedEvent (workflow_id, final_checkpoint_thread_ts). The status transition builds it and records the turn's final checkpoint in the callback context as ended_turn_checkpoint; the event is published after commit. CallbackWorker posts the answer from that checkpoint while the context still answers that turn (Workflow#callback_context_for_ended_turn).
  • One claim. The existing delivered_at claim dedups the turn event, the CI job-ended backstop and ServerSideTurnWorker, as on master. A failed post releases it for the Sidekiq retry.
  • Later turns post nothing. Workflow#current_turn_callback_context returns nil once the context's turn ended. This changes every CallbackWorker handler: with_adapter reads through it, so failure replies, approval and input cards stop for later turns, and so does ProgressDeliveryWorker. Before, an ended turn's context kept receiving them.
  • No per-turn reset. ExecuteRunService no longer resets the context's delivery state on each continuation (reset_messaging_turn! and per_turn_reset are gone), so resumes from the session page, MCP or the API leave the context as it is.
  • Stale backstops. The CI job-ended backstop acts only for the session's latest workload.
  • Delegated sessions still report to the parent's context as on master, even after the parent's turn ended: a child's result belongs to the turn that started it.
  • Turn identity is the latest top-level checkpoint when the session goes idle. That holds because the AI Gateway saves the turn's checkpoint before it reports IDLE.
Known limits
  • A turn that ends at input_required doesn't mark the context. Such a turn's answer is delivered through the detached Workhorse path or ServerSideTurnWorker. If the session is then continued from the session page, the next turn's approval card still goes to the original surface; its answer, progress and failures don't, because delivered_at is already set. On master the surface gets both the card and the answer. This needs a Slack session continued from the session page, and moves to the mailbox design (#633236 (closed)).
  • Slack live reads: the Slack adapter reads status_ts and the todo list live, so a late delivery after the next turn started would show the next turn's todos. Slack sessions don't rest idle in 19.6.
  • Duplicates: a crash between posting and recording delivered_at can post a turn twice. The retry budget is unchanged.
  • Mixed deploy: a WorkflowTurnEndedEvent job on a node still running the previous release raises InvalidEvent. This is harmless: only the flag produces idle.

References

How to set up and validate locally

developer/4.0.0 needs the chat engine, which isn't merged yet, so the AI Gateway runs a local branch with two open MRs. Sending a follow-up from the session page needs #633289, so example 2 starts the next turn from the Rails console instead, the way a client resume would.

Before you start

  • GDK with Duo Agent Platform working (Duo Workflow Service and AI Gateway running, a model available).
  • A GDK runner that picks up Duo Agent Platform jobs (Docker running).
  • A project where the Developer flow is turned on and its service account (for example duo-developer-<group>) is a member, and an issue in it.

Setup

  1. Check out this branch and restart Rails:

    git fetch origin duo/feature/633286-turn-ended-delivery
    git checkout duo/feature/633286-turn-ended-delivery
    gdk restart rails-web rails-background-jobs
  2. In your AI Gateway checkout (gdk/gitlab-ai-gateway), build a local branch from the chat engine (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7296) and developer/4.0.0 (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7519). The two MRs conflict only in tests, which -X ours settles:

    git fetch origin js-chat-engine-failure-rollback duo/feature/3173-developer-4-0-0-chat-partial
    git checkout -b local/633286-verify origin/js-chat-engine-failure-rollback
    git merge --no-edit -X ours origin/duo/feature/3173-developer-4-0-0-chat-partial

    Let the engine own developer/4.0.0 in duo_workflow_service/workflows/registry.py:

    -ENGINE_OWNED_CONFIG_VERSIONS: frozenset[tuple[str, str]] = frozenset()
    +ENGINE_OWNED_CONFIG_VERSIONS: frozenset[tuple[str, str]] = frozenset({("developer", "4.0.0")})

    Then restart it: gdk restart duo-workflow-service gitlab-ai-gateway.

  3. Turn the flag on for your user. If sessions fail with a 401 from a model provider you have no key for, also turn off model routing:

    bin/rails runner 'Feature.enable(:duo_developer_continuable_sessions, User.find_by_username("<you>"))'
    bin/rails runner 'Feature.disable(:duo_developer_model_routing)'
  4. Open a console with the GDK's token signing (source ../env.runit && bin/rails console) and define this helper. It prints the session's status, which turn its callback context answers, and the latest answer:

    def turn_state(id)
      w = Ai::DuoWorkflows::Workflow.find(id)
      answer = w.latest_ui_chat_log.reverse.find { |m| m['message_type'] == 'agent' && m['content'].present? }
      puts "status:  #{w.status_name}"
      puts "context: #{w.messaging_callback_context&.slice('note_id', 'ended_turn_checkpoint', 'delivered_at').inspect}"
      puts "answer:  #{answer&.dig('content').to_s.truncate(100)}"
    end

Example 1: a mention gets its answer when the session pauses

  1. Comment on the issue: @<developer service account> In one sentence: which port do the servers listen on? Don't change files.

    Expect: within 1–3 minutes the service account replies once in that thread, and the session (Automate > Sessions, newest entry) shows Idle. turn_state(<id>) prints status: idle and context: with your comment's note_id, an ended_turn_checkpoint and a delivered_at.

    On master, the session rests Idle and nothing is ever posted. That's the first bug.

Example 2: a follow-up without a surface stays in the session

  1. Start the next turn the way a client resume would, without a message from the thread:

    w = Ai::DuoWorkflows::Workflow.find(<id>)
    Current.organization = w.project.organization
    Gitlab::SafeRequestStore.ensure_request_store do
      Ai::DuoWorkflows::Runtimes::Ci.new(w, params_overrides: { goal: 'In one sentence, which language is the Ruby server written in? Do not change files.' }).execute
    end

    The request store keeps the service account's identity for the pipeline; without it the run fails with Insufficient permissions to create a new pipeline.

    Expect: within 1–3 minutes turn_state(<id>) prints status: idle with the new answer, and context: is unchanged. Nothing new appears in the thread. log/application_json.log has Duo Messaging: The session has moved past the turn its callback context answers for the session.

Example 3 (optional): a late delivery still posts the first answer, once Right after example 2, replay the first turn's end as if its delivery had never run:
w = Ai::DuoWorkflows::Workflow.find(<id>)
first_turn = w.messaging_callback_context['ended_turn_checkpoint']
event = Ai::DuoWorkflows::WorkflowTurnEndedEvent.build(workflow: w, final_checkpoint_thread_ts: first_turn)
w.release_messaging_callback_delivery! # models "the first delivery never ran"
Gitlab::SafeRequestStore.ensure_request_store do
  2.times { Ai::Messaging::CallbackWorker.new.handle_event(event) }
end

Expect: one new reply in the thread with the first answer (the port), not the second turn's. The second call logs Duo Messaging: Terminal outcome already delivered. The thread now shows the first answer twice, by construction.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Closes #633286

Merge request reports

Loading
Loading