[FF] dw_read_blobs_list — Reading internal checkpoint list from incremental blobs

Summary

Roll out the internal checkpoint list endpoint's checkpoint_headers reconstruction path, gated by the dw_read_blobs_list feature flag. Part of the incremental checkpoint read path (epic &22939).

This consumer is additionally gated by the master kill switch duo_workflow_read_incremental_checkpoints (604687) — it reconstructs from blobs only when both flags are on, and only for workflows with incremental_checkpoints_enabled set.

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

GET .../workflows/:id/checkpoints starts merging in checkpoint_headers rows that have no matching legacy checkpoints row, reconstructing channel_values from blobs for those entries. Two Duo Workflow Service consumers depend on this list staying complete: the STOP_RECOVERY boundary walk (checkpoints_reversed()) and the pre-18.8 GraphQL fallback (_fetch_most_recent_checkpoint()). If reconstruction is wrong, those paths could see corrupted or incomplete state on resume. Watch the AI Gateway / Duo Workflow Service error rate and the checkpoint reconstruction metrics/logs during rollout.

Rollout

Run all production /chatops in #production and cross-post to #g_agent-execution.

Non-production

/chatops gitlab run feature set dw_read_blobs_list 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set dw_read_blobs_list true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set dw_read_blobs_list <percentage> --actors

Or target specific actors instead:

/chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss dw_read_blobs_list true
/chatops gitlab run feature set --group=gitlab-org,gitlab-com dw_read_blobs_list true
/chatops gitlab run feature set --user=<gitlab-username-of-dri> dw_read_blobs_list true

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

  • Docs + version history updated
  • Breaking changes announced, if any
  • Change management issue opened, if required

Cleanup

Remove the flag once deemed stable. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check <merge-request-url> <milestone>
/chatops gitlab run feature delete dw_read_blobs_list --dev --pre --staging --staging-ref --production

Related