Load AI audit event details after the ClickHouse dedup

The agent artifacts pages no longer read each event's details payload while ClickHouse deduplicates and pages a session's audit events. details is loaded in a second, small query for only the rows that are returned.

Why

AuditEvents::AiAuditEvents::ClickHouseFinder#execute deduplicates ai_audit_events rows in an inner GROUP BY (a row can be ingested twice), then orders and pages in the outer query. The inner query carried any(details) for every event of the session. details is the JSON payload (prompts, tool arguments), several KB per event with no size cap for Duo events, so peak memory for one page of 20 events grew with the whole session, on a server other features share. Same pattern as the tool-call dedup in !265353 (merged).

Example, one session with 200,000 events of about 4 KB details: the page query peaked at 892 MiB before and 49 MiB after.

The change

  • ClickHouseFinder#execute leaves details out of the dedup and the page query.
  • New ClickHouseFinder.details_by_id(rows) and .with_details(rows): one SELECT id, details for the returned rows, filtered by their key columns and created_at range, 500 rows per query. Both export services use with_details, so an export still returns every event but ClickHouse holds at most 500 payloads at a time.
  • Types::AuditEvents::AiAuditEventType batch-loads details and humanAuthor for a page through BatchLoader::GraphQL, pooled per row kind (workflow or session) so a request runs at most two lookups, only when the client asks for those fields. Rows that already carry details resolve as before.
  • No GraphQL schema change, no migration, no UI change. No PostgreSQL query changes: PostgresqlFinder and the model are untouched, and the database label comes from the ee/app/finders/ path.

Measured

Local ClickHouse 26.9.2, synthetic data: one session with 200,000 events of about 4 KB details each (768 MiB of payload), first page of 21 events, peak memory from system.query_log.

Query Peak memory Read Time
Page query before (dedup carries details) 892 MiB 780 MiB 186 ms
Page query after 49 MiB 11 MiB 38 ms
details for the 21 returned rows 17 MiB 18 MiB 12 ms
details for 500 rows (one export batch) 22 MiB 18 MiB 13 ms
Queries and plan

ClickHouse page query (ClickHouseFinder#execute, a Duo session; an external session filters on ai_governance_session_id instead), before:

SELECT id, workflow_id, ai_governance_session_id, created_at, traversal_path, event_name, author_id, ip_address, details
FROM (
  SELECT id, workflow_id, any(ai_governance_session_id) AS ai_governance_session_id, created_at, traversal_path,
         any(event_name) AS event_name, any(author_id) AS author_id, any(ip_address) AS ip_address, any(details) AS details
  FROM ai_audit_events
  WHERE workflow_id = 12345 AND created_at >= '2026-09-30 12:00:00'
  GROUP BY traversal_path, workflow_id, created_at, id
) ai_audit_events
ORDER BY created_at DESC, id DESC LIMIT 21

After, details is gone from both the inner GROUP BY and the outer select:

SELECT id, workflow_id, ai_governance_session_id, created_at, traversal_path, event_name, author_id, ip_address
FROM (
  SELECT id, workflow_id, any(ai_governance_session_id) AS ai_governance_session_id, created_at, traversal_path,
         any(event_name) AS event_name, any(author_id) AS author_id, any(ip_address) AS ip_address
  FROM ai_audit_events
  WHERE workflow_id = 12345 AND created_at >= '2026-09-30 12:00:00'
  GROUP BY traversal_path, workflow_id, created_at, id
) ai_audit_events
ORDER BY created_at DESC, id DESC LIMIT 21

The returned rows then get one lookup per row kind (ClickHouseFinder.details_by_id), 500 rows per query:

SELECT id, details FROM ai_audit_events
WHERE traversal_path IN (...) AND workflow_id IN (...) AND ai_governance_session_id IN (...)
  AND id IN (...) AND created_at >= '<oldest returned>' AND created_at <= '<newest returned>'

EXPLAIN indexes = 1 for that lookup on a larger synthetic set (ClickHouse 26.9.2, 400,000 events, one session's first page of 21 ids): the partition key keeps 1 of 1 parts for the month, the created_at min-max index narrows to 11 of 82 granules, and the primary key (traversal_path, workflow_id, created_at, id) reads 1 of those 11, with details read only for that granule.

Decisions

  • A second query over a SQL-only rewrite: the page LIMIT is added by the GraphQL connection on the outer query and ClickHouse cannot push it through the GROUP BY, so any single-query shape (JOIN back, DISTINCT, LIMIT BY) still holds every payload while sorting.
  • Batch loading in the GraphQL type, over the pagination connection: it is the standard BatchLoader pattern, and there is no second query when details and humanAuthor are not requested.
  • Exports load details 500 rows at a time. A cap on the export size is a separate change.

Out of scope

Related to #635324.

Edited by Dheeraj Joshi

Merge request reports

Loading
Loading