Draft: Duo flow webhook callbacks via regular webhooks (duo_flow_callback_enabled flag + callback_hook_id)

Thomas Schmidtrequested to merge
duo/feature/flow-webhook-callbacks-rework-slice1 into master

What does this MR do and why?

Adds Duo flow webhook callbacks: a way for an external client to subscribe to a Duo flow's lifecycle over HTTP instead of polling. A regular project or group webhook is opted in as a callback endpoint via a new duo_flow_callback_enabled capability flag; the client then references that webhook by ID (callback_hook_id) when triggering a flow, and GitLab POSTs a small JSON payload on every state change (flow.started, flow.progress, flow.completed, flow.failed).

This reuses the existing GitLab webhook infrastructure end to end — encryption at rest, SSRF protection, delivery logging, retries, and payload signing — and surfaces callback hooks in the standard Settings > Webhooks UI and hooks REST API, so there is no separate webhook-management surface to learn.

Created as part of the DAP Extensibility challenge.

🎥 Demo: https://youtu.be/w2iPc4hjY74

How it works

  1. Enable once — create or edit a project or group webhook with duo_flow_callback_enabled: true (UI checkbox or hooks REST API). GitLab stores the URL + secret (encrypted) and returns the webhook id.
  2. Reference many times — pass callback_hook_id: <id> (and an optional client_reference) when creating a flow. Every flow started with that id delivers its lifecycle events to the endpoint.

A group webhook fires for any flow whose project is in that group or a descendant; a project webhook fires only for that project.

Changes

  • Migration — add duo_flow_callback_enabled boolean to web_hooks. It is a capability flag, not an event subscription: deliberately not named *_events and not registered in TriggerableHooks, so it never fires from event dispatch.
  • CallbackHooksFinder — resolves the ProjectHook/GroupHook records usable as a callback endpoint for a given project (its own project hooks plus hooks on the project's group and any ancestor group).
  • Hooks API + settings form — optional duo_flow_callback_enabled param on the project/group hooks REST API (params + entities), and a "Duo Agent Platform" checkbox in the webhook form that shows the hook ID once saved.
  • Flow triggering — build_messaging_callback_context resolves callback_hook_id via CallbackHooksFinder; an out-of-scope hook is rejected with 400. Adds an optional client_reference (max 255 chars) stored in the callback context and echoed in every payload so clients can correlate callbacks with their own records.
  • Delivery — a webhook messaging adapter builds the payload; WebhookDeliveryWorker sends it via WebHookService and raises on failure so Sidekiq retries with backoff, using a deterministic idempotency key per (workflow, event) for client-side de-duplication.
  • Docs — engineer integration guide (webhook_integration.md).

Payload shape

{
  "object_kind": "duo_workflow",
  "version": "1",
  "event": "flow.completed",
  "event_id": "flow.completed-2722",
  "project": {
    "id": 17,
    "path_with_namespace": "mygroup/myproject",
    "web_url": "https://gitlab.example.com/mygroup/myproject"
  },
  "workflow": {
    "id": 2722,
    "status": "finished",
    "web_url": "https://gitlab.example.com/mygroup/myproject/-/automate/agent-sessions/2722"
  },
  "client_reference": "run-abc123",
  "message": "Done!"
}

message is present on flow.completed (the agent's final answer); error is present on flow.failed. client_reference is present when supplied at trigger time.

Permissions

  • Configuring the flag: Maintainer+ on the project (project webhook) or Owner on the group (group webhook) — the standard admin_web_hook ability.
  • Triggering a flow with callback_hook_id: any user who can create the flow. The webhook was opted in by a Maintainer/Owner at configuration time, and scope containment (CallbackHooksFinder) is the authorization boundary — no separate admin_web_hook check is re-run at trigger time.

Design history / rationale (high-signal)

Paths deliberately not taken, recorded so we don't re-litigate them later:

  • A flag on a regular webhook, not a bespoke model + API. An earlier iteration used a dedicated FlowCallbackHook STI model and a /ai/duo_workflows/flow_callbacks CRUD API. Folding it into regular webhooks reuses encryption, SSRF protection, delivery logging, retries, and signing for free, avoids a parallel management surface, and makes callback hooks visible in the standard Webhooks UI/API. Trade-off: duo_flow_callback_enabled lives on the CE web_hooks table / project hooks API while the consuming feature is EE-only (see GA checklist).
  • Scoped to group/project, not organization. Anchoring to a group or project gives isolation on GitLab.com and a real permission model (admin_web_hook); an organization anchor gave neither.
  • A dedicated WebhookDeliveryWorker, not WebHook#async_execute. The standard async path (WebHookWorker) swallows HTTP failures and never redelivers — acceptable for event-stream webhooks that fire again on the next event, but a dropped terminal event (flow.completed) leaves a client hanging. So delivery runs WebHookService synchronously and raises on non-success to get Sidekiq retries + backoff, with a deterministic idempotency key for client-side dedup.

Commit structure (future split boundaries)

This MR is a reference for the full implementation and is intended to be split. Commits are organized along the likely slice boundaries:

  1. web_hooks column (migration)
  2. CallbackHooksFinder
  3. Hooks API + settings-form checkbox
  4. Flow-triggering wiring (callback_hook_id, client_reference)
  5. Delivery (payload adapter + WebhookDeliveryWorker)
  6. Integration guide docs

How to set up and validate locally (GDK)

Prereqs: an EE GDK with a license; Duo / agent platform usable in your instance; run gdk restart rails-web sidekiq after checkout so the new worker queue is picked up. Callback URLs must be public HTTPS (local/internal is SSRF-blocked) — use a https://webhook.site/<uuid> endpoint, or enable Admin > Settings > Network > Outbound requests > "Allow requests to the local network from webhooks and integrations".

  1. Enable the flag on a webhook (UI): Project (or Group) > Settings > Webhooks. Create a webhook pointing at your webhook.site URL, tick Duo Agent Platform > "Allow this webhook to be used as a Duo flow callback endpoint", and save. Note the Hook ID shown in that section.
    • Or via API: POST /api/v4/projects/:id/hooks with {"url": "https://webhook.site/<uuid>", "duo_flow_callback_enabled": true}.
  2. Trigger a flow referencing the hook id: 
    curl --request POST "http://gdk.test:3000/api/v4/ai/duo_workflows/workflows" \
  --header "PRIVATE-TOKEN: <api token>" --header "Content-Type: application/json" \
  --data '{"project_id": "<id>", "goal": "…", "workflow_definition": "developer/v1",
           "start_workflow": true, "environment": "web",
           "callback_hook_id": <hook id>, "client_reference": "run-1"}'
  3. Watch deliveries land on webhook.site (object_kind, version, project, client_reference, and the flow.* events). Delivery history is also logged under the webhook's Edit > Recent events.
  4. Verify scoping: referencing a hook that is not in scope for the flow's project (e.g. a project hook from a different project) is rejected with 400.

GA readiness checklist (known follow-ups)

Deliberately out of scope for this MR; tracked here so nothing is lost:

  • Feature flag for rollout (disabled by default, group actor), checked at trigger time in the workflows API.
  • License/tier gating: the checkbox and the callback_hook_id path should be gated on Duo Agent Platform availability for the namespace (Premium/Ultimate + Duo add-on). Currently the flag is visible/settable on any EE instance, and the param is declared in the CE project hooks API while the consuming feature is EE-only — decide whether to move it to an EE API extension.
  • "Test" button support: the webhook UI's Test dropdown fires sample payloads per trigger (via TestHooks::ProjectService / TestHooks::GroupService). Add a duo_flow_callback test trigger that sends a sample flow.completed payload so integrators can verify their endpoint + signature handling without running a real flow.
  • flow.progress opt-in per hook: progress frames contain agent reasoning/content and are more sensitive than lifecycle pings — consider a second checkbox instead of always-on.
  • Auto-disable & rate limiting verification: these now apply automatically (regular hooks), but the terminal-event path goes through WebhookDeliveryWorker/WebHookService directly rather than #async_execute — verify failure accounting feeds auto-disable as expected, and that per-plan webhook rate limits apply on this path.
  • Granular permission (optional): configuration reuses standard webhook permissions, which is fine; revisit only if product wants "may enable Duo callbacks" separated from "may manage webhooks".
  • User-facing docs: move/replace ee/app/services/ai/messaging/adapters/webhook_integration.md with real docs under doc/ (webhook events page + Duo Agent Platform page). (OpenAPI docs for the new hooks API field and callback_hook_id/client_reference params are already regenerated in this MR.)
  • Payload contract: version: "1" is in place; document the additive-only change policy.
  • Minor cleanup: CallbackHooksFinder.usable_for_project docstring describes a Relation/array — it returns the finder instance.
  • Audit events for enabling/disabling the flag (webhook update audit events may already cover this — verify).

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Thomas Schmidt

Merge request reports