Draft: Deliver each Duo turn's answer where the turn came from
What does this MR do and why?
When moving to the newly planned chat engine, Duo Developer sessions on developer/4.0.0 (the new flow we plan to run on that engine, behind the flag duo_developer_continuable_sessions) rest on the new idle status after each answer instead of finishing. With the work behind gitlab-org#23827 we are planning to allow continuing a session from the session sidebar as well.
Today an answer is posted to the thread only when a session finishes, so a resting session never posts its answer. And the callback context lives for the whole session, so a follow-up from the session page would post its answer into the original thread. For example, when Duo Developer was started by mentioning it in a comment, that comment would get a random reply from a later turn.
After this MR:
- A mention gets its answer as soon as the session pauses, not when it finishes.
- A thread gets the answer to the message it sent, and nothing else. Follow-ups from the session page, MCP or the Duo CLI show their answers in the session only. The thread gets no further answers, progress, approval cards or failure replies from those turns.
- A delayed or retried delivery still posts the right answer, once, even if the next turn has already started.
- One-shot flows are unchanged:
developer/3.0.0, code review, Slack assistant, webhooks and delegated sessions deliver as before.
How it works (for reviewers)
A callback context answers one turn: the one its surface's message started.- Turn ended event. Resting
idlepublishes the newWorkflowTurnEndedEvent(workflow_id,final_checkpoint_thread_ts). The status transition builds it and records the turn's final checkpoint in the callback context asended_turn_checkpoint; the event is published after commit.CallbackWorkerposts the answer from that checkpoint while the context still answers that turn (Workflow#callback_context_for_ended_turn). - One claim. The existing
delivered_atclaim dedups the turn event, the CI job-ended backstop andServerSideTurnWorker, as on master. A failed post releases it for the Sidekiq retry. - Later turns post nothing.
Workflow#current_turn_callback_contextreturns nil once the context's turn ended. This changes everyCallbackWorkerhandler:with_adapterreads through it, so failure replies, approval and input cards stop for later turns, and so doesProgressDeliveryWorker. Before, an ended turn's context kept receiving them. - No per-turn reset.
ExecuteRunServiceno longer resets the context's delivery state on each continuation (reset_messaging_turn!andper_turn_resetare gone), so resumes from the session page, MCP or the API leave the context as it is. - Stale backstops. The CI job-ended backstop acts only for the session's latest workload.
- Delegated sessions still report to the parent's context as on master, even after the parent's turn ended: a child's result belongs to the turn that started it.
- Turn identity is the latest top-level checkpoint when the session goes
idle. That holds because the AI Gateway saves the turn's checkpoint before it reportsIDLE.
Known limits
- A turn that ends at
input_requireddoesn't mark the context. Such a turn's answer is delivered through the detached Workhorse path orServerSideTurnWorker. If the session is then continued from the session page, the next turn's approval card still goes to the original surface; its answer, progress and failures don't, becausedelivered_atis already set. On master the surface gets both the card and the answer. This needs a Slack session continued from the session page, and moves to the mailbox design (#633236 (closed)). - Slack live reads: the Slack adapter reads
status_tsand the todo list live, so a late delivery after the next turn started would show the next turn's todos. Slack sessions don't restidlein 19.6. - Duplicates: a crash between posting and recording
delivered_atcan post a turn twice. The retry budget is unchanged. - Mixed deploy: a
WorkflowTurnEndedEventjob on a node still running the previous release raisesInvalidEvent. This is harmless: only the flag producesidle.
References
- #633286
- Spike: #633284 (closed)
- PoC: !261434 (closed)
- ADR-001: gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7419 (merged)
- Follow-ups through
ExecuteRunService: #633289, #628097 - Steering mailbox, where a surface's follow-ups and their delivery state will live: #633236 (closed)
- "Session finished" semantics: #633375
How to set up and validate locally
developer/4.0.0 needs the chat engine, which isn't merged yet, so the AI Gateway runs a local branch with two open MRs. Sending a follow-up from the session page needs #633289, so example 2 starts the next turn from the Rails console instead, the way a client resume would.
Before you start
- GDK with Duo Agent Platform working (Duo Workflow Service and AI Gateway running, a model available).
- A GDK runner that picks up Duo Agent Platform jobs (Docker running).
- A project where the Developer flow is turned on and its service account (for example
duo-developer-<group>) is a member, and an issue in it.
Setup
-
Check out this branch and restart Rails:
git fetch origin duo/feature/633286-turn-ended-delivery git checkout duo/feature/633286-turn-ended-delivery gdk restart rails-web rails-background-jobs -
In your AI Gateway checkout (
gdk/gitlab-ai-gateway), build a local branch from the chat engine (gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7296) anddeveloper/4.0.0(gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!7519). The two MRs conflict only in tests, which-X ourssettles:git fetch origin js-chat-engine-failure-rollback duo/feature/3173-developer-4-0-0-chat-partial git checkout -b local/633286-verify origin/js-chat-engine-failure-rollback git merge --no-edit -X ours origin/duo/feature/3173-developer-4-0-0-chat-partialLet the engine own
developer/4.0.0induo_workflow_service/workflows/registry.py:-ENGINE_OWNED_CONFIG_VERSIONS: frozenset[tuple[str, str]] = frozenset() +ENGINE_OWNED_CONFIG_VERSIONS: frozenset[tuple[str, str]] = frozenset({("developer", "4.0.0")})Then restart it:
gdk restart duo-workflow-service gitlab-ai-gateway. -
Turn the flag on for your user. If sessions fail with a 401 from a model provider you have no key for, also turn off model routing:
bin/rails runner 'Feature.enable(:duo_developer_continuable_sessions, User.find_by_username("<you>"))' bin/rails runner 'Feature.disable(:duo_developer_model_routing)' -
Open a console with the GDK's token signing (
source ../env.runit && bin/rails console) and define this helper. It prints the session's status, which turn its callback context answers, and the latest answer:def turn_state(id) w = Ai::DuoWorkflows::Workflow.find(id) answer = w.latest_ui_chat_log.reverse.find { |m| m['message_type'] == 'agent' && m['content'].present? } puts "status: #{w.status_name}" puts "context: #{w.messaging_callback_context&.slice('note_id', 'ended_turn_checkpoint', 'delivered_at').inspect}" puts "answer: #{answer&.dig('content').to_s.truncate(100)}" end
Example 1: a mention gets its answer when the session pauses
-
Comment on the issue:
@<developer service account> In one sentence: which port do the servers listen on? Don't change files.Expect: within 1–3 minutes the service account replies once in that thread, and the session (Automate > Sessions, newest entry) shows Idle.
turn_state(<id>)printsstatus: idleandcontext:with your comment'snote_id, anended_turn_checkpointand adelivered_at.On
master, the session rests Idle and nothing is ever posted. That's the first bug.
Example 2: a follow-up without a surface stays in the session
-
Start the next turn the way a client resume would, without a message from the thread:
w = Ai::DuoWorkflows::Workflow.find(<id>) Current.organization = w.project.organization Gitlab::SafeRequestStore.ensure_request_store do Ai::DuoWorkflows::Runtimes::Ci.new(w, params_overrides: { goal: 'In one sentence, which language is the Ruby server written in? Do not change files.' }).execute endThe request store keeps the service account's identity for the pipeline; without it the run fails with
Insufficient permissions to create a new pipeline.Expect: within 1–3 minutes
turn_state(<id>)printsstatus: idlewith the new answer, andcontext:is unchanged. Nothing new appears in the thread.log/application_json.loghasDuo Messaging: The session has moved past the turn its callback context answersfor the session.
Example 3 (optional): a late delivery still posts the first answer, once
Right after example 2, replay the first turn's end as if its delivery had never run:w = Ai::DuoWorkflows::Workflow.find(<id>)
first_turn = w.messaging_callback_context['ended_turn_checkpoint']
event = Ai::DuoWorkflows::WorkflowTurnEndedEvent.build(workflow: w, final_checkpoint_thread_ts: first_turn)
w.release_messaging_callback_delivery! # models "the first delivery never ran"
Gitlab::SafeRequestStore.ensure_request_store do
2.times { Ai::Messaging::CallbackWorker.new.handle_event(event) }
endExpect: one new reply in the thread with the first answer (the port), not the second turn's. The second call logs Duo Messaging: Terminal outcome already delivered. The thread now shows the first answer twice, by construction.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
Closes #633286