## Problem `GitLabWorkflow.aput()` (`duo_workflow_service/checkpointer/gitlab_workflow.py`) receives a `new_versions: ChannelVersions` parameter that tells it exactly which state channels changed in the current step, but **ignores it entirely** and always serializes and POSTs the full `checkpoint["channel_values"]` — the entire workflow state. For a workflow mid-run with a long conversation history, this means every single LangGraph step (even one that only updated `status`) POSTs the full message history, plan, and UI logs. Compare with how `MemorySaver` handles this: ```python # MemorySaver.put — incremental values = c.pop("channel_values") for k, v in new_versions.items(): self.blobs[(thread_id, ns, k, v)] = self.serde.dumps_typed(values[k]) # only stores channels that changed this step ``` vs current `GitLabWorkflow.aput()`: ```python payload = { "checkpoint": checkpoint, # full channel_values every time ... } await self._client.apost(...) # new_versions ignored ``` ## Proposed solution 1. **`aput`**: Strip `channel_values` from the checkpoint to only the keys present in `new_versions`. Store these as channel blobs alongside the checkpoint header (id, versions, metadata). 2. **Rails API**: Accept channel blobs keyed by `(channel, version)` on write. Return all blobs needed to reconstruct state on read. 3. **`aget_tuple`**: Reconstruct full state by walking the checkpoint chain (via `parent_ts`) and merging blob layers — same reconstruction pattern `MemorySaver` uses in memory. Infrastructure for per-channel writes already exists: `checkpoint_writes_batch` endpoint (`gitlab_workflow.py:844`) stores interrupt writes per channel. The blob pattern is the same concept applied to every step. ## Impact - Payload per `aput` call becomes proportional to **what changed** in a step, not total conversation length - For a step that only updates `status`, the payload shrinks from \~full state to \~a few bytes - Combines well with async writes: small payloads + non-blocking = minimal overhead per step ## Notes - Read path becomes a merge over the checkpoint chain instead of a single fetch — one-time cost per session resume, acceptable trade-off - No change to data residency: checkpoint blobs still go to SM customer's GitLab instance - `langgraph-checkpoint 2.1.2` is already in use; `new_versions` is already passed by LangGraph on every `aput` call ## Feature flags & rollout Every flag is `type: wip`, `default_enabled: false`, `group::agent execution`. Fill the last two columns as each flag is turned on. | Flag | Gates | MR | Rollout issue | Enabled for `gitlab-org` | Enabled globally | |------|-------|----|---------------|:------------------------:|:----------------:| | `duo_workflow_incremental_checkpoints` | Global switch: Shadow-writes blobs + slim header alongside the full checkpoint (dual write) | on master (phase 1) | gitlab-org/gitlab#604684 | :rocket: | :rocket: | | `duo_workflow_write_incremental_only` | Stops shadow-writing the full checkpoint — headers + blobs only | gitlab-org/gitlab!247367 | gitlab-org/gitlab#607008 | | | | `duo_workflow_read_incremental_checkpoints` | Master off-switch for all blob reads; every read consumer also requires this | gitlab-org/gitlab!247133 | gitlab-org/gitlab#604687 | :rocket: | | | `dw_read_blobs_api` | Internal checkpoint API / gateway (`by_thread_ts`) | gitlab-org/gitlab!247134 | gitlab-org/gitlab#606987 | | | | `dw_read_blobs_trace` | Trace download (`trace.jsonl`) | gitlab-org/gitlab!247135 | gitlab-org/gitlab#606988 | :rocket: | | | `dw_read_blobs_graphql` | GraphQL checkpoint fields (`duoMessages`, `lastDuoMessage`, `executionStatus`, raw `checkpoint`) | gitlab-org/gitlab!243813 | gitlab-org/gitlab#606989 | | | | `dw_read_blobs_notifications` | Messaging / notifications (callback worker, progress reader, result email) | gitlab-org/gitlab!247369 | gitlab-org/gitlab#606990 | | | ### Rollout procedure Order is dependency-driven: reads must be fully on blobs **before** the write side stops emitting the full checkpoint. 1. **Write blobs (dual write)** — `duo_workflow_incremental_checkpoints`. Blobs + slim headers accumulate next to the full checkpoint. Must reach global before read rollout so every workflow has blobs to read from. 2. **Read kill switch** — enable `duo_workflow_read_incremental_checkpoints`. On its own this changes nothing (each consumer also needs its own flag); it is the master off-switch for the whole read path. 3. **Read consumers, one at a time** — for each `dw_read_blobs_*`, enable for `gitlab-org`, validate dashboards, then go global. A consumer reconstructs from blobs only when the kill switch **and** its own flag are on, so consumers roll out and roll back independently. 4. **Stop the dual write** — only after **all** read consumers are global and stable (blobs are the sole read source), enable `duo_workflow_write_incremental_only` for `gitlab-org`, then global. This drops the legacy full-checkpoint write. Per flag — `gitlab-org` first, then global: ``` /chatops gitlab run feature set --group=gitlab-org <flag> true # gitlab-org /chatops gitlab run feature set <flag> true # global ``` Rollback a single consumer: `/chatops gitlab run feature set <flag> false`. Disable the entire read path at once: `/chatops gitlab run feature set duo_workflow_read_incremental_checkpoints false`.