Duo Workflow: failed checkpoint POST silently advances the incremental delta baseline, leaving a permanent hole in blob history

Summary

A failed checkpoint POST does not stop the workflow. The gateway updates its in-memory delta baseline before the POST completes, and it never checks the response status. Later checkpoints then build deltas against a checkpoint that was never saved.

Failure scenario

aput in duo_workflow_service/checkpointer/gitlab_workflow.py updates state.prev_channel_values and state.prev_checkpoint_id (around line 1579) before it sends the checkpoint POST (around line 1611). It never checks the response status. The status only feeds a metric label.

ExecutorGitLabHttpClient._call (duo_workflow_service/gitlab/executor_http_client.py:295-310) returns a failed response after retries instead of raising. Non-retryable statuses, such as 400 or 413, return right away.

If one POST fails, LangGraph keeps running. The next aput sees a parent_ts that matches the cached prev_checkpoint_id, so the stale-cache guard (around line 1549) does not trigger. Every later blob is a delta against a checkpoint that does not exist.

Legacy full-snapshot checkpoints recovered from this: the next snapshot restated the whole state. Incremental blobs do not restate anything, so the gap is permanent.

Rails makes this worse. Ai::DuoWorkflows::Workflow#walk_ancestry (ee/app/models/ai/duo_workflows/workflow.rb, around line 1220) stops silently when a parent header is missing; only cycles raise. With duo_workflow_write_incremental_only on, there is no legacy row to fall back to, so the reconstructed state collapses to the newest deltas only.

Proposed fix

Gateway: raise on a non-2xx checkpoint POST, or force the next write to be a group start that re-sends every channel in full, so the chain re-anchors.

Rails: make a missing intermediate ancestor loud (log or raise) instead of silently truncating the walk.

Fix this before duo_workflow_write_incremental_only rolls out.

Context

Found while reviewing size-limit hardening in ai-assist MR gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!6528 (merged). Code references are against gateway main at commit a327213ae and gitlab-org/gitlab master.