Read getUserAgentFlows messages from incremental checkpoint blobs
What does this MR do and why?
Follow-up to !242398 (closed) (phase 2 read path). Makes the getUserAgentFlows
GraphQL query read session messages from incremental checkpoint blobs
instead of the full checkpoint header, and adds a cheap lastDuoMessage
field for the session-list preview.
The session list surfaces each session's latest message through
latestCheckpoint, resolved by WorkflowCheckpointEventPresenter. Reading
ui_chat_log from the full checkpoint header loaded the entire
channel_values JSONB per row — the data-load pattern that forced the revert
in !241288 (merged) (incident:
gitlab-com/gl-infra/gitlab-dedicated/incident-management#3702).
Changes
duo_messagesreads from blobs behind the per-consumerdw_read_blobs_graphqlflag plusreconstruct_from_blobs?(theduo_workflow_read_incremental_checkpointskill switch + the workflow'sincremental_checkpoints_enabledcolumn) — gated in the presenter byreconstruct_from_blobs_for_graphql?. Header read is kept otherwise.- Channel-scoped read —
Workflow#reconstructed_channel(checkpoint, channel)decodes only theui_chat_logblobs, not every channel (conversation_history is usually the bulk). The reconstructor now folds append/dict deltas in place (concat) instead of+, which was O(n²). lastDuoMessagefield — a singleDuoMessagethat reads only the newestui_chat_logblob (Workflow#latest_channel_delta).duoMessagesis unchanged; the list can switch to this field.
Memory measurements (real 97-message session, per workflow row)
Measured with memory_profiler (DB load + transform):
| read path | total/row | vs header |
|---|---|---|
| header read (current) | ~1.45 MB | 1× |
full reconstructed_channel_values |
~4.8 MB | 3.3× |
channel-scoped ui_chat_log (this MR) |
~1.85 MB | 1.3× |
lastDuoMessage (this MR) |
~0.25 MB | 0.17× |
Full reconstruction is far heavier than the header read; channel-scoping cuts
that ~60% but still can't beat a single JSONB read for message-heavy sessions
(decoding many compressed blobs is allocation-heavy). lastDuoMessage is ~6×
lighter than the header read, so the reverted list preview can be re-enabled
without the original data load.
What this does NOT fix: the N+1
This MR reduces the per-row cost, not the number of queries. latestCheckpoint
is still resolved per workflow row (workflow.checkpoints.latest), and
latest_channel_delta / reconstructed_channel add a blob query per row. A page
of N sessions is therefore still ~2N queries — the same N+1 the reverted feature
had. lastDuoMessage only makes each query cheap and small (a single LIMIT 1
row instead of a ~296 KB JSONB header or ~88 blob rows), shrinking the blast
radius rather than removing it.
Eliminating the N+1 is separate work: batch-load the latest checkpoint and its
newest ui_chat_log blob for the whole page (e.g. GraphQL BatchLoader /
preloading), so the list resolves in a constant number of queries. Tracked as
follow-up; out of scope here.
Notes
- The checkpoint header still embeds full
channel_valuestoday (dual-write); the blob read becomes the sole option once the write path stops embedding it. - Frontend is unchanged. Re-enabling the list preview requires the fragment to
request
latestCheckpoint { lastDuoMessage { ... } }— deferred to the frontend follow-up.
Related
- Base MR: !242398 (closed) · Reverted feature: !241288 (merged)
- Work item: #604677 (closed)
Restructure — epic &22939
Part of the phase-2 read-path epic. Each consumer is gated by its own flag in addition to the master kill switch.
Code gates on the per-consumer
dw_read_blobs_graphqlflag in addition to the sharedduo_workflow_read_incremental_checkpointskill switch, viaWorkflowCheckpointEventPresenter#reconstruct_from_blobs_for_graphql?.