Persist channel membership on the checkpoint header

Problem

Duo Workflow checkpoints are stored incrementally: a slim header row in p_duo_workflows_checkpoint_headers plus per-channel blobs in p_duo_workflows_checkpoint_blobs.

The blobs are an append-only log per channel. There is no way to record that a channel was deleted, so rebuilding a checkpoint gives the union of every channel ever written, not the channels that checkpoint actually had. LangGraph deletes channels on most steps (apply_writes clears armed ephemeral channels), so stale keys pile up.

Until now the full checkpoint row in p_duo_workflows_checkpoints carried the real membership. Under the feature flag duo_workflow_write_incremental_only that row is no longer written, so the membership is lost.

channel_versions (already on the header) can't stand in. It keeps versions for channels that were consumed and no longer hold a value.

QA saw this directly: the reconstructed channel_values dropped LangGraph's branch:to:* channels, so a resumed graph saw no armed tasks and stopped silently. See !247134 (comment 3652137773)

What this MR does

  • Adds a channel_keys text[] column to p_duo_workflows_checkpoint_headers (range-partitioned by workflow_created_at). The column is nullable with no default. NULL means the row was written before the column existed. The read path treats that as "membership not recorded".
  • Ai::DuoWorkflows::CreateCheckpointService#write_checkpoint_header now stores checkpoint['channel_values'].keys on the header. The keys are already in hand at write time, so this adds no extra query and no API change.
  • Adds specs in ee/spec/services/ai/duo_workflows/create_checkpoint_service_spec.rb and ee/spec/requests/api/ai/duo_workflows/workflows_internal_spec.rb.

Not in this MR

  • The read/reconstruction side that selects by these keys: !247134 (merged)
  • The AI gateway change to blob every channel regardless of type (drop the scalar allowlist), so scalar channels like branch:to:* have a blob behind the key.

Database

No query plans. This MR adds no new queries. The column is only written, by an INSERT that already runs. Adding a nullable column with no default is a catalog-only change: it takes a brief ACCESS EXCLUSIVE lock on the parent and each partition, and rewrites nothing.

The column carries a CHECK (cardinality(channel_keys) <= 100) constraint. A checkpoint's channel set is its flow state keys plus one branch:to:<node> routing channel per node armed for the next step, so the count follows the flow's node count. Local data over 200 checkpoints shows 12 keys at most; the largest shipped flow builds about 20 nodes, so 100 leaves room for larger custom flows. NULL rows stay legal, because cardinality(NULL) is NULL.

Size: p_duo_workflows_checkpoint_headers was created in 19.2 (20260710130000) and is written only for workflows where duo_workflow_incremental_checkpoints is enabled. It is range-partitioned daily by workflow_created_at and the partition manager drops partitions older than 30 days, so the row count is bounded by a 30-day window rather than growing without limit. Existing rows gain no bytes, because NULL is stored in the row's null bitmap. New rows carry one short text array (one entry per live channel, typically under 20 entries of ~10-25 characters), so a few hundred bytes per row at most.

db:gitlabcom-database-testing was triggered for this migration and reports the production table size and the migration timings against a GitLab.com clone.

Migration up:

main: == 20260814092215 AddChannelKeysToDuoWorkflowsCheckpointHeaders: migrating ====
main: -- add_column(:p_duo_workflows_checkpoint_headers, :channel_keys, :text, {:array=>true})
main:    -> 0.0616s
main: == 20260814092215 AddChannelKeysToDuoWorkflowsCheckpointHeaders: migrated (0.0674s)

Migration down:

main: == 20260814092215 AddChannelKeysToDuoWorkflowsCheckpointHeaders: reverting ====
main: -- remove_column(:p_duo_workflows_checkpoint_headers, :channel_keys, :text, {:array=>true})
main: == 20260814092215 AddChannelKeysToDuoWorkflowsCheckpointHeaders: reverted (0.0394s)

References

Edited by Eduardo Bonet

Merge request reports

Loading
Loading