Store a slim checkpoint header instead of the full checkpoint

Why

p_duo_workflows_checkpoints was designed for the full-checkpoint model: the whole state lives in the row, partitioned by created_at. Those assumptions fight the incremental design, and we keep working around them. The partition problem (#605054 (closed)) is the clearest example — we're bending created_at/partitioning just to make incremental reads prune.

Rather than carry that debt forward, design a table for the incremental scenario from the start. The checkpoint row becomes a header: only what's needed to rebuild a CheckpointTuple and isn't already in blobs; channel_values is reconstructed from blobs on read (that path exists).

(Double-writing today is fine — that's the shadow-write transition. Old rows age out via the 30-day TTL.)

Current design Proposed design
image image

Proposed table: p_duo_workflow_checkpoint_headers

workflow_id, project_id/namespace_id (sharding key)
workflow_created_at   -- partition key
created_at, updated_at
thread_ts, parent_ts
current_thread
checkpoint  -- header only: channel_versions, versions_seen, v, ts, pending_sends
metadata    -- langgraph metadata

Partitioned by workflow_created_at (dedicated column) so reads prune to one partition and created_at/updated_at stay honest — same scheme as the blobs change in #605054 (closed).

Dropped vs. today: channel_values (→ blobs), ui_chat_log column (dead — it's a channel). Kept (needed to resume, not in blobs): the checkpoint header + metadata.

Prerequisite: self-contained blob groups (AIGW)

Blobs are currently a partial delta layer on top of the full checkpoint. On read we seed from the header, then overlay blob deltas. A compaction bump only re-seeds the one channel that compacted; other channels keep appending in the new group with no base. Drop the header and those channels lose their history.

Fix: compaction should start a new group across all channels (it's per-channel today) — re-seed every channel as a full snapshot at group start. Then each group reconstructs on its own. (status is already always-blobbed; other scalars are safe to drop.)

Readers to move to reconstruction

Done in !243813 (merged): duoMessages / lastDuoMessage, trace. Still reading channel_values directly:

  • WorkflowCheckpointEventPresenter#execution_status (status)
  • Checkpoint#ui_chat_log → ProgressReader, CallbackWorker
  • workflow email
  • write-time goal backfill (pairs with the AIGW change)

Sequence

  1. AIGW: cross-channel compaction (self-contained groups)
  2. New p_duo_workflow_checkpoint_headers; route writes/reads behind the existing flag
  3. Migrate readers → drop channel_values

Blobs workflow_created_at partitioning lands first in #605054 (closed).

Edited by 🤖 GitLab Bot 🤖