## Why `GitLabWorkflow.aput()` currently ignores the `new_versions` parameter that LangGraph passes on every step and always POSTs the **full** `checkpoint["channel_values"]` to the Rails API. For a workflow mid-run with a long conversation history, this means every step — even one that only updates `status` — sends the entire message history, plan, and UI logs over the wire. A POC measured the impact on real workflows: | | Full values | Incremental deltas | Saving | |-|-------------|-------------------|--------| | `conversation_history` total bytes | 1,434,536 B | 121,984 B | **91.5%** | | All channels total bytes | 1,822,452 B | 466,380 B | **74.4%** | The savings compound with conversation length: the longer the workflow runs, the larger the gap between full-state and incremental payloads. ## What — Phase 1: shadow writes Phase 1 introduces the full infrastructure for incremental blob storage while keeping the **read path completely unchanged**. The gateway continues to embed full `channel_values` in every checkpoint, so all existing Rails instances remain compatible. Blobs are written alongside the checkpoint as a shadow — they accumulate in the DB but nothing consumes them yet. This makes phase 1 safe to ship and roll back independently. The work spans four MRs across two repositories: - **[!239245](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239245) — DB layer**: introduces the range-partitioned `p_duo_workflows_checkpoint_blobs` table, the `CheckpointBlob` model, the `current_thread` column on `p_duo_workflows_checkpoints`, and the `duo_workflow_incremental_checkpoints` feature flag (default off). Rows are dropped automatically after 30 days via partition drop. - **[!239247](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239247) — API + service**: `CreateCheckpointService` bulk-inserts per-channel blobs on write; `Workflow#accumulated_blobs_for` reads blobs for the current `current_thread` group ordered by `id`; the `CheckpointBlob` entity base64-encodes `data` for the gateway. - **[!239249](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239249) — Capability advertisement**: wires `incremental_checkpoints` through the workhorse session negotiation (`runner.go`) and the frontend client (`constants.js`), so Rails can signal to the gateway that it supports the feature. - **[ai-assist!5672](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/5672) — Gateway shadow writes**: `aput()` sends `channel_blobs` alongside the full `compressed_checkpoint`. The gateway only sends blobs when the Rails instance declares the `incremental_checkpoints` capability, so older instances are never affected. ## Shadow write design and reasoning The core idea is that `aput()` sends **both** the full checkpoint (as today) **and** a set of per-channel blobs derived from `new_versions`. Rails stores the blobs but the read path (`aget_tuple`) still reconstructs state from the full checkpoint. This gives us: 1. **Zero risk to existing workflows** — the read path is untouched; a rollback of the feature flag reverts to the pre-phase-1 behaviour with no data loss. 2. **Real production data to validate phase 2** — blobs accumulate in the DB under real traffic, so the read-path reconstruction logic in phase 2 can be validated against actual data before the full checkpoint is removed from the payload. 3. **Capability negotiation** — the gateway only sends blobs when Rails advertises `incremental_checkpoints` in the session capabilities. This means the gateway MR can ship before the Rails MR without any coordination risk. **What gets blobed and what doesn't.** Only list and dict-of-list channels are included in blobs (e.g. `conversation_history`, `ui_chat_log`). Scalar channels (`status`, `goal`, `project`, etc.) are excluded — they are tiny, always full replacements, and always recoverable from the full checkpoint. This keeps blob volume proportional to the channels that actually grow with conversation length. **Delta logic.** Two helpers compute the minimal payload per channel: - `_list_delta(prev, current)` — returns the tail slice `val[prev_len:]` for pure appends, the full value for shrinks/rewrites, or `None` if unchanged. - `_dict_of_list_delta(prev, current)` — applies `_list_delta` per key for dict-of-list channels; unchanged keys are omitted entirely. Each blob carries a `step_action` field — `"conversation"` for append deltas, `"compaction"` for full replacements — which is the **authoritative** append-vs-replace signal for Rails on the read side. **`current_thread` grouping.** The gateway tracks a `current_thread` counter that starts at 0 and increments each time a compaction is detected (any list channel shrinks). Rails stores this on both the checkpoint and each blob, enabling `accumulated_blobs_for` to read only blobs from the current compaction generation with a single indexed query — no recursive `parent_ts` walk needed. The counter resets to 0 on gateway restart, so it is a **grouping hint only**; `step_action` per blob is derived from actual channel values and is always the authoritative compaction signal. **Why not LangGraph's native blob storage?** LangGraph's Postgres/SQLite savers write a full snapshot per `(channel, version)` — not a list-append delta. For `conversation_history` at step 50, the blob would still contain all 50 entries. The `_list_delta` / `_dict_of_list_delta` logic that produces the 91.5% saving on `conversation_history` is domain-specific and has no equivalent in the LangGraph API. Additionally, `GitLabWorkflow` is a proxy with no local storage — every read and write goes through the Rails HTTP API — so adopting LangGraph's blob format would still require Rails to implement the same storage and return blobs on retrieval, which is exactly what phase 2 does. ## Feature flag The entire write path is gated behind `duo_workflow_incremental_checkpoints` (default off). The flag controls: - Whether Rails stores incoming `channel_blobs` from the gateway - Whether Rails advertises the `incremental_checkpoints` capability in session negotiation (which in turn controls whether the gateway sends blobs at all) Enabling the flag on a subset of users or projects allows measuring blob volume and storage overhead before committing to the full read-path switch in phase 2.