Dedup incremental checkpoint blobs for redundant delta sends

Context

Incremental checkpoint blobs (p_duo_workflows_checkpoint_blobs) were introduced in gitlab-org/gitlab!239247 behind duo_workflow_incremental_checkpoints. The write path stores per-step channel deltas; current_thread groups them and step_action (conversation = append delta, compaction = full replacement) is the authoritative fold signal.

That MR does not dedup blobs, which is safe today because:

  • The gateway sends each delta exactly once.
  • The read/reconstruction fold path is not live yet (no duo_workflow_read_incremental_checkpoints consumer in Rails).

So duplicates can currently be neither produced nor misread.

Why we need dedup

We plan to send the same delta multiple times to add redundancy against failed requests (see Alper's thread on !239247). Once that lands, reconstruction must not double-apply a duplicated append delta.

The current unique index idx_duo_wf_checkpoint_blobs_unique on (project_id, workflow_id, thread_ts, channel, version, created_at) does not dedup retried sends, because created_at is set to Time.current per request (and created_at is forced into the key by the table being PARTITION BY RANGE (created_at)).

Proposed approach

Write-side (storage / idempotent insert):

  • Make created_at deterministic per logical delta (e.g. derived from thread_ts) so retries land in the same partition with an identical key.
  • Switch bulk_insert! → insert_all(..., on_duplicate: :skip) keyed on (workflow_id, thread_ts, channel, version, step_action).

Read-side (correctness):

  • Make the reconstruction fold idempotent: order by version, dedup, and apply compaction-precedence — for a given (channel, version), a compaction (full value) overrides a conversation (tail delta).

Important caveat

Dedup must key on (channel, version, step_action) (or a content hash), not (channel, version) alone. The same version integer can legitimately be emitted with different data/step_action: a normal append emits step_action="conversation" with only the appended tail, while a force_rewrite (after a gateway restart that loses _prev_channel_values) re-emits the same version as step_action="compaction" with the full value. These are not interchangeable — see _serialize_channel_blobs in gitlab-ai-gateway.

Sequencing constraint

This is a safe follow-up only if it lands before either of these goes live:

  1. the gateway's redundant-send behaviour, and
  2. the read/reconstruction fold path (duo_workflow_read_incremental_checkpoints).

Whichever ships first is the deadline.

Open question to confirm during implementation

Can a restart re-emit an already-persisted version as a compaction (vs. only ever emitting versions strictly greater than what's stored)? This determines whether the dual-representation collision is real or only theoretical, and thus whether read-side compaction-precedence is strictly required.

Edited by 🤖 GitLab Bot 🤖