Add p_duo_workflows_checkpoint_headers table

What does this MR do?

Adds p_duo_workflows_checkpoint_headers: a slim per-checkpoint "header" table for the incremental Duo Workflow checkpoint design, plus a shadow write from CreateCheckpointService.

A header row stores only what is needed to rebuild a CheckpointTuple and is not already in the blobs: the langgraph checkpoint minus channel_values (channel_versions, versions_seen, v, ts, pending_sends) plus its metadata. channel_values is reconstructed from p_duo_workflows_checkpoint_blobs on read (that path already exists).

Writes are double-written during the shadow-write transition: the full p_duo_workflows_checkpoints row and the new header are both written when incremental checkpoints are enabled on the workflow. Old checkpoint rows age out via the 30-day TTL. No reads move in this MR — moving readers off channel_values and dropping it is a follow-up (issue sequence step 3).

Part of #605653 (closed). Mirrors the workflow_created_at partitioning scheme from !244403 (merged) (blobs); that MR is a reference, not a dependency — this table is independent and targets master.

Table design

  • Range-partitioned by a dedicated workflow_created_at column (= the workflow's created_at, written on every row) so all of a workflow's headers share one daily partition and a lookup equality-prunes to it, while created_at/updated_at stay honest.
  • Daily partitions dropped past CHECKPOINT_RETENTION_DAYS (30) enforce the TTL — cleanup is a partition DROP, not a row DELETE.
  • Sharding key project_id/namespace_id (exactly one non-null, check constraint). Partitioned FK on workflow_idduo_workflows_workflows (ON DELETE CASCADE); the sharding keys intentionally have no FK (partition drop handles cleanup), allowlisted in spec/db/schema_spec.rb.
  • Append-only (no unique index), like p_duo_workflows_checkpoints: a re-sent checkpoint writes another row and readers take the latest. A non-unique (workflow_id, thread_ts) index serves the "latest header for a thread" lookup.
  • checkpoint/metadata jsonb have no JsonSchemaValidator (the langgraph checkpoint shape is externally defined) — allowlisted alongside the sibling Ai::DuoWorkflows::Checkpoint.
CREATE TABLE p_duo_workflows_checkpoint_headers (
    id bigint NOT NULL,
    workflow_id bigint NOT NULL,
    project_id bigint,
    namespace_id bigint,
    workflow_created_at timestamp with time zone NOT NULL,
    created_at timestamp with time zone NOT NULL,
    updated_at timestamp with time zone NOT NULL,
    current_thread integer DEFAULT 0 NOT NULL,
    checkpoint jsonb NOT NULL,
    metadata jsonb NOT NULL,
    thread_ts text NOT NULL,
    parent_ts text,
    CONSTRAINT check_duo_wf_checkpoint_headers_sharding_key CHECK ((num_nonnulls(namespace_id, project_id) = 1))
    -- + thread_ts/parent_ts length checks
)
PARTITION BY RANGE (workflow_created_at);

Database

gitlab_main_org. New empty table; no data migration. ~database review requested.

Anticipated growth: one row per checkpoint (≈ one per workflow step). Transient — every row is dropped 30 days after its workflow started (partition DROP). Bounded by active-workflow volume over a rolling 30-day window; same order as the sibling p_duo_workflows_checkpoints, expected small.

Access patterns:

  • Write: one INSERT per checkpoint (via bulk_insert!, dedup-skip), inside the existing checkpoint-create transaction, only when incremental checkpoints are enabled.
  • Read: none yet (follow-up). Future reads are keyed by workflow_id + workflow_created_at, pruning to a single daily partition.
Migration up
== 20260710130000 CreatePDuoWorkflowsCheckpointHeaders: migrating =============
== 20260710130000 CreatePDuoWorkflowsCheckpointHeaders: migrated (0.0340s) ====
== 20260710130001 AddWorkflowForeignKeyToPDuoWorkflowsCheckpointHeaders: migrating
== 20260710130001 AddWorkflowForeignKeyToPDuoWorkflowsCheckpointHeaders: migrated (0.5070s)
== 20260710130002 EnsureIdUniquenessForPDuoWorkflowsCheckpointHeaders: migrating
== 20260710130002 EnsureIdUniquenessForPDuoWorkflowsCheckpointHeaders: migrated (0.0382s)
Migration down
== 20260710130002 EnsureIdUniquenessForPDuoWorkflowsCheckpointHeaders: reverting
== 20260710130002 EnsureIdUniquenessForPDuoWorkflowsCheckpointHeaders: reverted (0.0494s)
== 20260710130001 AddWorkflowForeignKeyToPDuoWorkflowsCheckpointHeaders: reverting
== 20260710130001 AddWorkflowForeignKeyToPDuoWorkflowsCheckpointHeaders: reverted (0.1498s)
== 20260710130000 CreatePDuoWorkflowsCheckpointHeaders: reverting =============
== 20260710130000 CreatePDuoWorkflowsCheckpointHeaders: reverted (0.0516s) ====

Test coverage

  • CheckpointHeader model spec (associations, validations, sharding-key sync).
  • CreateCheckpointService spec: header shadow-write with channel_values stripped, workflow_created_at anchoring, append-only re-send, no write when incremental disabled.

Queries

The only write is the shadow header insert in CreateCheckpointService (bulk_insert! of a single row, append-only, inside the existing checkpoint-create transaction, when incremental checkpoints are enabled):

INSERT INTO "p_duo_workflows_checkpoint_headers"
  ("workflow_id","project_id","namespace_id","workflow_created_at",
   "created_at","updated_at","current_thread","checkpoint","metadata",
   "thread_ts","parent_ts")
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11);

Single-row INSERT (the id is assigned by the table's BEFORE INSERT trigger); no scan, no dedup/ON CONFLICT. There are no reads on the new table in this MR.

The header is append-only (no unique index): a re-sent checkpoint writes another row and readers take the latest, matching p_duo_workflows_checkpoints.

Edited by Eduardo Bonet

Merge request reports

Loading