Draft: POC: Incremental checkpoint blob storage [skip ci]

Eduardo Bonetrequested to merge
596714-poc-incremental-checkpoints into master

What

POC for Implement incremental checkpoints in GitLabWorkflow checkpoint saver — Rails API side.

Companion to the ai-gateway POC: gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!5650 (closed)

Changes

New table: duo_workflows_checkpoint_blobs

Stores per-channel versioned blobs keyed by (workflow_id, thread_ts, channel, version). Each blob is a base64-encoded serialized channel value — only changed channels are stored per step.

Write path

POST /checkpoints accepts an optional channel_blobs array:

{ "channel": "messages", "version": "2", "write_type": "msgpack", "data": "<base64>" }

Blobs are bulk-inserted atomically with the checkpoint record in a single transaction.

Read path

GET /checkpoints/:id walks the parent_ts chain and returns accumulated blobs oldest-first in checkpoint_blobs. The gateway merges them per-channel to reconstruct full channel_values without a separate backfill.

Rollout strategy (phased)

Phase Gateway writes Gateway reads Rails stores
1 — Shadow full checkpoint + blobs old full checkpoint both
2 — Live full checkpoint + blobs blobs both
3 — Cleanup blobs only blobs both (old becomes minimal)

No backfill needed — checkpoints have a 30-day TTL.

Notes

  • This is a POC: no feature flag, no partitioning, no full test coverage
  • write_type column name matches the gateway payload exactly (avoids Rails STI type conflict)
  • Chain-walking on read is N+1 queries for N steps — acceptable for POC; will use a CTE in production

References

Merge request reports