Deliver Duo flow progress to callback webhooks
What does this MR do and why?
Adds flow.progress to Duo flow webhook callbacks, so a subscribed client sees the agent's current plan and activity while the flow runs instead of only the start and finish pings.
Split out of !249145 (merged) so the delivery contract — and the data it exposes — can be reviewed on its own.
Progress is delivered differently from the terminal events, on purpose:
flow.started / completed / failed |
flow.progress |
|
|---|---|---|
| Path | WebhookDeliveryWorker |
inline via hook.execute |
| On failure | raises, Sidekiq retries | tracked and swallowed |
| Why | a dropped flow.completed leaves a client waiting forever |
the next checkpoint supersedes a lost frame, so a retry would deliver stale state |
Each frame gets its own event_id, so a client does not mistake a new frame for a redelivery. Terminal events keep a stable event_id across retries, which is what makes those dedupable.
supports_live_progress? is the switch: CreateCheckpointService only schedules the debounced ProgressDeliveryWorker for adapters that opt in. Before this MR the webhook adapter returned false, so no progress jobs were scheduled at all.
This MR also changes the Slack adapter. latest_todos and latest_monologue moved out of Ai::Messaging::Adapters::Slack into a shared Ai::Messaging::UiChatLog that both adapters now include, so the two readers of the ui_chat_log format cannot drift apart as it evolves. latest_monologue became latest_agent_message(messages, limit:), because each surface keeps its own truncation limit — both 1500 today, free to move apart. Slack's behaviour is unchanged: same parsing, same limit. It is a shipping code path, though, so the diff is worth a look. This came out of review on this MR.
Base#on_progress gains a workflow: argument, which adapters serving many flows through one endpoint need to attribute a frame. It defaults to nil and Slack ignores it, so the change is backward compatible.
Payload adds plan (the agent's latest todo list) and activity (its latest message, truncated at 1500 chars). Both are omitted when empty. No new queue and no new flag. Progress is governed like the terminal events: duo_flow_callback_hooks decides which flows may subscribe, and the delivery path reads only the per-hook duo_flow_callback_enabled switch, never the flag. A frame is a single primary key lookup on web_hooks, so unticking a hook stops a running flow's frames.
Worth explicit review: data exposure
Progress frames carry agent reasoning and tool arguments, which is more sensitive than a lifecycle ping. For the experimental release this is on for any hook with duo_flow_callback_enabled, gated per root namespace by duo_flow_callback_hooks.
The alternative is a second per-hook opt-in so a hook can take lifecycle events without progress. That needs another column, so it is deliberately deferred to GA (&22923) rather than decided quietly here.
Where this fits
Part of Duo flow webhook callbacks — letting an external client subscribe to a Duo flow's lifecycle over a normal webhook instead of polling.
Stacked on !249145 (merged) (adapter and delivery worker). !249147 (merged) and !249149 (merged) branch off the same parent, so this MR does not depend on their code. The duo_flow_callback_hooks flag is defined in !249147 (merged) though, so the feature needs that MR too.
References
- Issue: #606804 (closed)
- Reference implementation this was split from (closed PoC): !246849 (closed)
How to set up and validate locally
Needs !249147 (merged) too, since progress only flows for a workflow that has a callback context.
Feature.enable(:duo_flow_callback_hooks, Group.find_by_full_path('my-group').root_ancestor)- Create a project or group webhook pointing at a public HTTPS endpoint with Duo flow callbacks enabled.
- Trigger a flow with
callback_hook_id: <hook id>. - Watch
flow.progressframes arrive while the agent runs — at most one every two seconds, since delivery is debounced per workflow — each carrying the currentplanandactivity. - Confirm frames stop once the flow is terminal and
flow.completedarrives.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.