Draft: Remove the incremental checkpoint read gates

Eduardo Bonetrequested to merge
629088-remove-the-incremental-checkpoint-read-gates into master

What does this MR do and why?

Removes the six blob-read feature flags (duo_workflow_read_incremental_checkpoints and dw_read_blobs_{api,graphql,trace,notifications,list}), all fully rolled out on GitLab.com. Gitlab::DuoWorkflow::IncrementalBlobGate collapses to one #read?, gated on the workflow's incremental_checkpoints_enabled column and the newest checkpoint header's channel membership. The gate class stays until the column drops in #627332. Second step after !255931 (merged) flips the same six flags to default-on; this MR stays Draft until that one has merged and the AI Gateway change is deployed.

Design notes
  • IncrementalBlobGate#read? (ee/lib/gitlab/duo_workflow/incremental_blob_gate.rb) replaces for_notifications?, for_list?, for_trace?, for_graphql?, graphql_candidate?, and reconstruct?. It's true when incremental_checkpoints_enabled is set and legacy_checkpoint_fallback? is false for the newest header.
  • Workflow.prime_latest_checkpoint_headers / prime_earliest_checkpoint_headers (ee/app/models/ai/duo_workflows/workflow.rb) check workflow.incremental_checkpoints_enabled? directly, not the gate, so filtering a page of workflows doesn't trigger a header query. Same for the __start__ blob read in ee/app/services/ai/duo_workflows/create_checkpoint_service.rb (write path). Behaviour is unchanged at both call sites.
  • API::Helpers::DuoWorkflowHelpers#push_feature_flags no longer pushes dw_read_blobs_api or duo_workflow_read_incremental_checkpoints to the AI Gateway.
  • GraphQL field descriptions on Query.duoWorkflowBranches and DuoMessage.threadTs/parentTs/alternativeCount no longer mention the flag. doc/api/graphql/reference/_index.md and public/-/graphql/introspection_result.json are regenerated.
  • The trace endpoint keeps its read=incremental|legacy override and legacy branch until #611971.
  • Specs: contexts that stubbed a read flag off became "workflow not created with incremental checkpoints" (update!(incremental_checkpoints_enabled: false)). The callback-hook spec in ee/spec/requests/api/ai/duo_workflows/workflows_spec.rb stubs the write flag duo_workflow_incremental_checkpoints: false instead, so the created workflow stays on the plain checkpoint table. Tests that only asserted flag-off behaviour are deleted.
  • Deleted the six YAML files under ee/config/feature_flags/wip/. Changelog: other, EE: true.
  • Ordering: the AI Gateway MR gitlab-org/modelops/applied-ml/code-suggestions/ai-assist!6941 (merged) must merge and deploy first. Today the gateway gates its by_thread_ts read on the two pushed flags plus the column; once this MR deletes the definitions Rails can't push them, and the gateway would fall back to the paged legacy read, which is empty under write-only.
  • This MR must merge before #629089 (stop legacy checkpoint writes). On self-managed both flag families default off, so stopping legacy writes while reads still hit the legacy table would lose data.

References

Screenshots or screen recordings

No UI change.

How to set up and validate locally

  1. Start the GDK at http://gdk.test:3000 with a project that has Duo Agent Platform enabled.
  2. In the rails console, run Feature::Definition.get(:dw_read_blobs_graphql). Expect: nil.
  3. Start a Duo Agentic Chat session in the project with any prompt. Expect: a workflow with incremental_checkpoints_enabled: true that writes headers and blobs.
  4. Query GraphQL: duoWorkflowWorkflows(projectPath:) { nodes { id latestCheckpoint { threadTs duoMessages { content threadTs parentTs } } } }. Expect: messages come back with threadTs set (blob read path).
  5. Call GET http://gdk.test:3000/api/v4/ai/duo_workflows/workflows/<id>/trace.jsonl with a PAT. Expect: ui_chat_log lines with the chat content.
  6. In the rails console, run Ai::DuoWorkflows::Workflow.find(<id>).incremental_blob_gate.read?. Expect: true.
  7. On that workflow, run update!(incremental_checkpoints_enabled: false) and call incremental_blob_gate.read? on a fresh find. Expect: false.
Extra checks
  • Call the trace endpoint with ?read=legacy. Expect: it reads from the checkpoints table.
  • A workflow whose newest header has channel_keys: nil. Expect: read? returns false and readers use the legacy table.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Eduardo Bonet

Merge request reports