Add api for Duo flow callback hooks

Carla Dragorequested to merge
607028-add-webhook-adapter into master

What does this MR do?

Adds the registration surface for Duo flow callback hooks: a client (for example, AutoFlow) registers an HTTPS callback endpoint and signing secret once, and gets back an ID it can later reference (callback_hook_id) when triggering a Duo flow, instead of polling for completion.

This MR only adds registration (model + API + authorization). Delivering flow lifecycle events to a registered hook, and wiring callback_hook_id into the create-flow endpoint, will be in a separate follow-up MR.

New API

POST   /ai/duo_workflows/flow_callbacks
GET    /ai/duo_workflows/flow_callbacks
GET    /ai/duo_workflows/flow_callbacks/:id
DELETE /ai/duo_workflows/flow_callbacks/:id
  • Ai::DuoWorkflows::FlowCallbackHook is a WebHook STI subtype, organization-scoped (mirrors SystemHook). The callback URL and secrets are encrypted at rest via the existing WebHook/attr_encrypted infrastructure and are never returned by the API.
  • Registration is authorized via three new, dedicated abilities (create_duo_flow_callback_hook, read_duo_flow_callback_hook, delete_duo_flow_callback_hook), gated to organization owners/admins in EE::Organizations::OrganizationPolicy — the same tier already used for the other CD/Duo organization-scoped abilities in that policy, and consistent with how project/group webhook administration requires Maintainer+ rather than the broader read-access needed to trigger a Duo workflow.
  • Endpoints are wired for granular access tokens (route_setting :authorization), with corresponding raw and assignable permission definitions under config/authz/.

Design decisions worth calling out

  • No feature flag. The change is purely additive (existing create-flow behavior is untouched until the follow-up MR adds an opt-in param), the endpoints are gated by a dedicated, narrowly-scoped ability, and nothing can trigger this code path yet (no client is wired to use it). If an emergency kill switch is ever needed once delivery lands, the generic drop_sidekiq_jobs_<WorkerName> / run_sidekiq_jobs_<WorkerName> operational flags cover that with zero additional code, per-worker, for free.
  • Rate limiting and auto-disable for the delivery worker are intentionally out of scope here and tracked separately (see References) — they are the right per-hook mitigations for a misbehaving endpoint, rather than something this MR needs to solve.

Query plan

Ai::DuoWorkflows::FlowCallbackHook is STI on the existing web_hooks table, so this adds no migration, but two code paths triggered Danger's database-review check: the for_organization scope (ee/app/models/ai/duo_workflows/flow_callback_hook.rb:29) and the destroy! call in the delete endpoint (ee/lib/api/ai/duo_workflows/flow_callbacks.rb:146). Both resolve to simple, indexed single-table queries against web_hooks.

1. for_organization scope (list/get/delete lookups)

Generated SQL for the paginated list endpoint (flow_callback_hooks.recent_first, first page):

SELECT "web_hooks".* FROM "web_hooks"
WHERE "web_hooks"."type" = 'Ai::DuoWorkflows::FlowCallbackHook'
  AND "web_hooks"."organization_id" = $1
ORDER BY "web_hooks"."id" DESC
LIMIT 20 OFFSET 0

EXPLAIN (ANALYZE, BUFFERS) (run locally against representative seed data — 5 organizations, ~18 FlowCallbackHook rows total, then rolled back):

Limit  (cost=2.18..2.18 rows=1 width=572) (actual time=0.009..0.010 rows=6 loops=1)
  Buffers: shared hit=5
  ->  Sort  (cost=2.18..2.18 rows=1 width=572) (actual time=0.009..0.010 rows=6 loops=1)
        Sort Key: id DESC
        Sort Method: quicksort  Memory: 28kB
        Buffers: shared hit=5
        ->  Index Scan using index_web_hooks_on_type on web_hooks  (cost=0.14..2.17 rows=1 width=572) (actual time=0.003..0.005 rows=6 loops=1)
              Index Cond: ((type)::text = 'Ai::DuoWorkflows::FlowCallbackHook'::text)
              Filter: (organization_id = 1007)
              Rows Removed by Filter: 12
              Buffers: shared hit=2
Planning:
  Buffers: shared hit=121
Planning Time: 0.169 ms
Execution Time: 0.019 ms

web_hooks already has both index_web_hooks_on_type and index_web_hooks_on_organization_id (btree, non-partial). At this tiny scale the planner picked the type index (cost is effectively tied between the two); as the number of FlowCallbackHook rows per organization grows relative to other hook types, it can equally pick index_web_hooks_on_organization_id instead — either way this resolves via a single index scan, never a sequential scan, and rows per organization are expected to stay very small (a handful of registered callback endpoints per org). No new index is needed.

2. destroy! (delete endpoint)

DELETE FROM "web_hooks" WHERE "web_hooks"."id" = $1

EXPLAIN (ANALYZE, BUFFERS):

Delete on web_hooks  (cost=0.14..2.16 rows=0 width=0) (actual time=0.008..0.008 rows=0 loops=1)
  Buffers: shared hit=6
  ->  Index Scan using web_hooks_pkey on web_hooks  (cost=0.14..2.16 rows=1 width=6) (actual time=0.005..0.005 rows=1 loops=1)
        Index Cond: (id = 19)
        Buffers: shared hit=5
Planning:
  Buffers: shared hit=2
Planning Time: 0.023 ms
Execution Time: 0.079 ms

Single-row delete by primary key. FlowCallbackHook#web_hook_logs has no dependent: option (matching SystemHook), so destroy! does not cascade to web_hook_logs and issues no additional queries.

How to set up and validate locally

curl --request POST \
  --header "PRIVATE-TOKEN: <your_access_token>" \
  --header "Content-Type: application/json" \
  --data '{"url": "https://example.com/duo/callbacks", "name": "Test"}' \
  --url "http://gdk.test:3000/api/v4/ai/duo_workflows/flow_callbacks"

Requires the calling user to be an Owner of the current organization.

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #607028

Edited by Carla Drago

Merge request reports