[FF] duo_workflow_write_incremental_only -- switch checkpoint writes to incremental-only

Summary

Roll out the feature currently behind the duo_workflow_write_incremental_only feature flag.

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

This flag stops the shadow-write of the legacy full-state checkpoint row (p_duo_workflows_checkpoints); only the slim header (p_duo_workflows_checkpoint_headers) and blobs (p_duo_workflows_checkpoint_blobs) are written.

Precondition: enable this flag only after the incremental-checkpoint read path is live for the same actors. Consumers that still read the full row (GraphQL workflow_events_updated, trace.jsonl, latest-checkpoint lookups) will see gaps for checkpoints written while the flag is on, because those rows are no longer persisted. The flag requires duo_workflow_incremental_checkpoints (blobs are only written when incremental checkpoints are enabled).

Additional confirmed consumer: the internal checkpoints list/get REST endpoints in ee/lib/api/ai/duo_workflows/workflows_internal.rb also read full checkpoint rows with no header/blob reconstruction. The Duo Workflow Service depends on them for STOP_RECOVERY resume (checkpoints_reversed()), LangGraph time-travel/replay (aget_tuple with a checkpoint_id), and a fallback for GitLab versions before 18.8 (_fetch_most_recent_checkpoint()). MR gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!6363 (merged) (open, unmerged) covers only the aget_tuple/checkpoint_id case, and only when both DW_READ_BLOBS_API and the workflow's incremental_checkpoints_enabled column are on. The STOP_RECOVERY boundary walk (checkpoints_reversed()) and the pre-18.8 fallback have no fix in flight. See #612586 (closed) for full details.

Watch database size for p_duo_workflows_checkpoints — it should stop growing / begin to shrink (30-day TTL) once the flag is on.

Rollout

Run all production /chatops in #production and cross-post the results to the team channel.

Non-production

/chatops gitlab run feature set duo_workflow_write_incremental_only 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set duo_workflow_write_incremental_only true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set duo_workflow_write_incremental_only <percentage> --actors

Or target specific actors instead:

/chatops gitlab run feature set --group=gitlab-org,gitlab-com duo_workflow_write_incremental_only true
/chatops gitlab run feature set --user=eduardobonet duo_workflow_write_incremental_only true

Before global rollout

  • Incremental-checkpoint read path is live for the actors being enabled.
  • duo_workflow_incremental_checkpoints is enabled for the same actors.
  • Database size dashboards confirmed as the source of truth for the expected decrease.

Cleanup

Remove the flag once deemed stable. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check <merge-request-url> <milestone>
/chatops gitlab run feature delete duo_workflow_write_incremental_only --dev --pre --staging --staging-ref --production

Related

Edited by Eduardo Bonet