Add p_duo_workflows_checkpoint_headers table
What does this MR do?
Adds p_duo_workflows_checkpoint_headers: a slim per-checkpoint "header"
table for the incremental Duo Workflow checkpoint design, plus a shadow write
from CreateCheckpointService.
A header row stores only what is needed to rebuild a CheckpointTuple and is
not already in the blobs: the langgraph checkpoint minus channel_values
(channel_versions, versions_seen, v, ts, pending_sends) plus its
metadata. channel_values is reconstructed from
p_duo_workflows_checkpoint_blobs on read (that path already exists).
Writes are double-written during the shadow-write transition: the full
p_duo_workflows_checkpoints row and the new header are both written when
incremental checkpoints are enabled on the workflow. Old checkpoint rows age
out via the 30-day TTL. No reads move in this MR — moving readers off
channel_values and dropping it is a follow-up (issue sequence step 3).
Part of #605653 (closed). Mirrors the workflow_created_at
partitioning scheme from !244403 (merged) (blobs); that MR is a
reference, not a dependency — this table is independent and targets
master.
Table design
- Range-partitioned by a dedicated
workflow_created_atcolumn (= the workflow'screated_at, written on every row) so all of a workflow's headers share one daily partition and a lookup equality-prunes to it, whilecreated_at/updated_atstay honest. - Daily partitions dropped past
CHECKPOINT_RETENTION_DAYS(30) enforce the TTL — cleanup is a partition DROP, not a row DELETE. - Sharding key
project_id/namespace_id(exactly one non-null, check constraint). Partitioned FK onworkflow_id→duo_workflows_workflows(ON DELETE CASCADE); the sharding keys intentionally have no FK (partition drop handles cleanup), allowlisted inspec/db/schema_spec.rb. - Append-only (no unique index), like
p_duo_workflows_checkpoints: a re-sent checkpoint writes another row and readers take the latest. A non-unique(workflow_id, thread_ts)index serves the "latest header for a thread" lookup. checkpoint/metadatajsonb have noJsonSchemaValidator(the langgraph checkpoint shape is externally defined) — allowlisted alongside the siblingAi::DuoWorkflows::Checkpoint.
CREATE TABLE p_duo_workflows_checkpoint_headers (
id bigint NOT NULL,
workflow_id bigint NOT NULL,
project_id bigint,
namespace_id bigint,
workflow_created_at timestamp with time zone NOT NULL,
created_at timestamp with time zone NOT NULL,
updated_at timestamp with time zone NOT NULL,
current_thread integer DEFAULT 0 NOT NULL,
checkpoint jsonb NOT NULL,
metadata jsonb NOT NULL,
thread_ts text NOT NULL,
parent_ts text,
CONSTRAINT check_duo_wf_checkpoint_headers_sharding_key CHECK ((num_nonnulls(namespace_id, project_id) = 1))
-- + thread_ts/parent_ts length checks
)
PARTITION BY RANGE (workflow_created_at);Database
gitlab_main_org. New empty table; no data migration. ~database review requested.
Anticipated growth: one row per checkpoint (≈ one per workflow step). Transient
— every row is dropped 30 days after its workflow started (partition DROP). Bounded
by active-workflow volume over a rolling 30-day window; same order as the sibling
p_duo_workflows_checkpoints, expected small.
Access patterns:
- Write: one INSERT per checkpoint (via
bulk_insert!, dedup-skip), inside the existing checkpoint-create transaction, only when incremental checkpoints are enabled. - Read: none yet (follow-up). Future reads are keyed by
workflow_id+workflow_created_at, pruning to a single daily partition.
Migration up
== 20260710130000 CreatePDuoWorkflowsCheckpointHeaders: migrating =============
== 20260710130000 CreatePDuoWorkflowsCheckpointHeaders: migrated (0.0340s) ====
== 20260710130001 AddWorkflowForeignKeyToPDuoWorkflowsCheckpointHeaders: migrating
== 20260710130001 AddWorkflowForeignKeyToPDuoWorkflowsCheckpointHeaders: migrated (0.5070s)
== 20260710130002 EnsureIdUniquenessForPDuoWorkflowsCheckpointHeaders: migrating
== 20260710130002 EnsureIdUniquenessForPDuoWorkflowsCheckpointHeaders: migrated (0.0382s)Migration down
== 20260710130002 EnsureIdUniquenessForPDuoWorkflowsCheckpointHeaders: reverting
== 20260710130002 EnsureIdUniquenessForPDuoWorkflowsCheckpointHeaders: reverted (0.0494s)
== 20260710130001 AddWorkflowForeignKeyToPDuoWorkflowsCheckpointHeaders: reverting
== 20260710130001 AddWorkflowForeignKeyToPDuoWorkflowsCheckpointHeaders: reverted (0.1498s)
== 20260710130000 CreatePDuoWorkflowsCheckpointHeaders: reverting =============
== 20260710130000 CreatePDuoWorkflowsCheckpointHeaders: reverted (0.0516s) ====Test coverage
CheckpointHeadermodel spec (associations, validations, sharding-key sync).CreateCheckpointServicespec: header shadow-write withchannel_valuesstripped,workflow_created_atanchoring, append-only re-send, no write when incremental disabled.
Queries
The only write is the shadow header insert in CreateCheckpointService
(bulk_insert! of a single row, append-only, inside the existing
checkpoint-create transaction, when incremental checkpoints are enabled):
INSERT INTO "p_duo_workflows_checkpoint_headers"
("workflow_id","project_id","namespace_id","workflow_created_at",
"created_at","updated_at","current_thread","checkpoint","metadata",
"thread_ts","parent_ts")
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11);Single-row INSERT (the id is assigned by the table's BEFORE INSERT trigger);
no scan, no dedup/ON CONFLICT. There are no reads on the new table in this MR.
The header is append-only (no unique index): a re-sent checkpoint writes
another row and readers take the latest, matching p_duo_workflows_checkpoints.