Load AI audit event details after the ClickHouse dedup
The agent artifacts pages no longer read each event's details payload while ClickHouse deduplicates and pages a session's audit events. details is loaded in a second, small query for only the rows that are returned.
Why
AuditEvents::AiAuditEvents::ClickHouseFinder#execute deduplicates ai_audit_events rows in an inner GROUP BY (a row can be ingested twice), then orders and pages in the outer query. The inner query carried any(details) for every event of the session. details is the JSON payload (prompts, tool arguments), several KB per event with no size cap for Duo events, so peak memory for one page of 20 events grew with the whole session, on a server other features share. Same pattern as the tool-call dedup in !265353 (merged).
Example, one session with 200,000 events of about 4 KB details: the page query peaked at 892 MiB before and 49 MiB after.
The change
ClickHouseFinder#executeleavesdetailsout of the dedup and the page query.- New
ClickHouseFinder.details_by_id(rows)and.with_details(rows): oneSELECT id, detailsfor the returned rows, filtered by their key columns andcreated_atrange, 500 rows per query. Both export services usewith_details, so an export still returns every event but ClickHouse holds at most 500 payloads at a time. Types::AuditEvents::AiAuditEventTypebatch-loadsdetailsandhumanAuthorfor a page throughBatchLoader::GraphQL, pooled per row kind (workflow or session) so a request runs at most two lookups, only when the client asks for those fields. Rows that already carrydetailsresolve as before.- No GraphQL schema change, no migration, no UI change. No PostgreSQL query changes:
PostgresqlFinderand the model are untouched, and thedatabaselabel comes from theee/app/finders/path.
Measured
Local ClickHouse 26.9.2, synthetic data: one session with 200,000 events of about 4 KB details each (768 MiB of payload), first page of 21 events, peak memory from system.query_log.
| Query | Peak memory | Read | Time |
|---|---|---|---|
Page query before (dedup carries details) |
892 MiB | 780 MiB | 186 ms |
| Page query after | 49 MiB | 11 MiB | 38 ms |
details for the 21 returned rows |
17 MiB | 18 MiB | 12 ms |
details for 500 rows (one export batch) |
22 MiB | 18 MiB | 13 ms |
Queries and plan
ClickHouse page query (ClickHouseFinder#execute, a Duo session; an external session filters on ai_governance_session_id instead), before:
SELECT id, workflow_id, ai_governance_session_id, created_at, traversal_path, event_name, author_id, ip_address, details
FROM (
SELECT id, workflow_id, any(ai_governance_session_id) AS ai_governance_session_id, created_at, traversal_path,
any(event_name) AS event_name, any(author_id) AS author_id, any(ip_address) AS ip_address, any(details) AS details
FROM ai_audit_events
WHERE workflow_id = 12345 AND created_at >= '2026-09-30 12:00:00'
GROUP BY traversal_path, workflow_id, created_at, id
) ai_audit_events
ORDER BY created_at DESC, id DESC LIMIT 21After, details is gone from both the inner GROUP BY and the outer select:
SELECT id, workflow_id, ai_governance_session_id, created_at, traversal_path, event_name, author_id, ip_address
FROM (
SELECT id, workflow_id, any(ai_governance_session_id) AS ai_governance_session_id, created_at, traversal_path,
any(event_name) AS event_name, any(author_id) AS author_id, any(ip_address) AS ip_address
FROM ai_audit_events
WHERE workflow_id = 12345 AND created_at >= '2026-09-30 12:00:00'
GROUP BY traversal_path, workflow_id, created_at, id
) ai_audit_events
ORDER BY created_at DESC, id DESC LIMIT 21The returned rows then get one lookup per row kind (ClickHouseFinder.details_by_id), 500 rows per query:
SELECT id, details FROM ai_audit_events
WHERE traversal_path IN (...) AND workflow_id IN (...) AND ai_governance_session_id IN (...)
AND id IN (...) AND created_at >= '<oldest returned>' AND created_at <= '<newest returned>'EXPLAIN indexes = 1 for that lookup on a larger synthetic set (ClickHouse 26.9.2, 400,000 events, one session's first page of 21 ids): the partition key keeps 1 of 1 parts for the month, the created_at min-max index narrows to 11 of 82 granules, and the primary key (traversal_path, workflow_id, created_at, id) reads 1 of those 11, with details read only for that granule.
Decisions
- A second query over a SQL-only rewrite: the page LIMIT is added by the GraphQL connection on the outer query and ClickHouse cannot push it through the GROUP BY, so any single-query shape (JOIN back, DISTINCT, LIMIT BY) still holds every payload while sorting.
- Batch loading in the GraphQL type, over the pagination connection: it is the standard BatchLoader pattern, and there is no second query when
detailsandhumanAuthorare not requested. - Exports load details 500 rows at a time. A cap on the export size is a separate change.
Out of scope
- The session artifact download cap is Cap the AI session artifact download and say wh... (!266536) (stacked on this MR). The default window on
duoWorkflowSessionArtifactsand the one-session-per-request limit on itsauditEventsare Bound Duo session artifact listings to a 180-da... (!265854 - merged) and Limit legacy session artifact auditEvents to on... (!265860 - merged).
Related to #635324.