Read getUserAgentFlows messages from incremental checkpoint blobs

What does this MR do and why?

Follow-up to !242398 (closed) (phase 2 read path). Makes the getUserAgentFlows GraphQL query read session messages from incremental checkpoint blobs instead of the full checkpoint header, and adds a cheap lastDuoMessage field for the session-list preview.

The session list surfaces each session's latest message through latestCheckpoint, resolved by WorkflowCheckpointEventPresenter. Reading ui_chat_log from the full checkpoint header loaded the entire channel_values JSONB per row — the data-load pattern that forced the revert in !241288 (merged) (incident: gitlab-com/gl-infra/gitlab-dedicated/incident-management#3702).

Changes

  1. duo_messages reads from blobs behind the per-consumer dw_read_blobs_graphql flag plus reconstruct_from_blobs? (the duo_workflow_read_incremental_checkpoints kill switch + the workflow's incremental_checkpoints_enabled column) — gated in the presenter by reconstruct_from_blobs_for_graphql?. Header read is kept otherwise.
  2. Channel-scoped readWorkflow#reconstructed_channel(checkpoint, channel) decodes only the ui_chat_log blobs, not every channel (conversation_history is usually the bulk). The reconstructor now folds append/dict deltas in place (concat) instead of +, which was O(n²).
  3. lastDuoMessage field — a single DuoMessage that reads only the newest ui_chat_log blob (Workflow#latest_channel_delta). duoMessages is unchanged; the list can switch to this field.

Memory measurements (real 97-message session, per workflow row)

Measured with memory_profiler (DB load + transform):

read path total/row vs header
header read (current) ~1.45 MB
full reconstructed_channel_values ~4.8 MB 3.3×
channel-scoped ui_chat_log (this MR) ~1.85 MB 1.3×
lastDuoMessage (this MR) ~0.25 MB 0.17×

Full reconstruction is far heavier than the header read; channel-scoping cuts that ~60% but still can't beat a single JSONB read for message-heavy sessions (decoding many compressed blobs is allocation-heavy). lastDuoMessage is ~6× lighter than the header read, so the reverted list preview can be re-enabled without the original data load.

What this does NOT fix: the N+1

This MR reduces the per-row cost, not the number of queries. latestCheckpoint is still resolved per workflow row (workflow.checkpoints.latest), and latest_channel_delta / reconstructed_channel add a blob query per row. A page of N sessions is therefore still ~2N queries — the same N+1 the reverted feature had. lastDuoMessage only makes each query cheap and small (a single LIMIT 1 row instead of a ~296 KB JSONB header or ~88 blob rows), shrinking the blast radius rather than removing it.

Eliminating the N+1 is separate work: batch-load the latest checkpoint and its newest ui_chat_log blob for the whole page (e.g. GraphQL BatchLoader / preloading), so the list resolves in a constant number of queries. Tracked as follow-up; out of scope here.

Notes

  • The checkpoint header still embeds full channel_values today (dual-write); the blob read becomes the sole option once the write path stops embedding it.
  • Frontend is unchanged. Re-enabling the list preview requires the fragment to request latestCheckpoint { lastDuoMessage { ... } } — deferred to the frontend follow-up.

Restructure — epic &22939

Part of the phase-2 read-path epic. Each consumer is gated by its own flag in addition to the master kill switch.

  • Delivery issue: #607000
  • Feature flag: dw_read_blobs_graphql
  • FF rollout issue: #606989

Code gates on the per-consumer dw_read_blobs_graphql flag in addition to the shared duo_workflow_read_incremental_checkpoints kill switch, via WorkflowCheckpointEventPresenter#reconstruct_from_blobs_for_graphql?.

Edited by Eduardo Bonet

Merge request reports

Loading