Bound duoWorkflowEvents page cost behind a feature flag
What does this MR do and why?
duoWorkflowEvents rebuilds each row's checkpoint from its whole blob chain, separately for every row, and each rebuilt row carries nearly the full session state. The cost of a page grows with the number of rows times the session length. On a long session, a first: 100 request can run past the 30-second GraphQL timeout and return tens of megabytes.
Behind the limit_duo_workflow_events_page_cost flag (gitlab_com_derisk, off by default, checked against the current user), this MR:
- Caps
Query.duoWorkflowEventsandProject.duoWorkflowEventsat 5 rows per page. Largerfirstvalues are clamped, not rejected, so paging withafterstill reaches the whole history. The default page size stays at 1. - Builds
checkpointandcompressedCheckpointfor a page from one blob query with a shared decode cache, the way the internal workflows API already does. Selecting both fields rebuilds each checkpoint once.
The batching is on the DuoWorkflowEvent type, so the flag also changes how checkpoint and compressedCheckpoint are built under firstCheckpoint, latestCheckpoint, and the workflowEventsUpdated subscription. Those return one event per workflow, so they still make one blob query per workflow, as before.
Who reads checkpoint:
-
The language server, which the IDE extensions and the Duo CLI use, reads it in three operations:
getDuoWorkflowEventsandgetLatestCheckpointinqueries.ts. The IDE's agentic chat uses both, and the Duo CLI usesgetLatestCheckpoint.- The
workflowEventsUpdatedsubscription inworkflow_events_response_channel.ts, which streams live session updates to the IDE.
None of them pass
first, so the cap doesn't change what they receive. The flag only changes how theircheckpointis built, and the result is the same. -
The web UI only reads
latestCheckpoint { duoMessages }, which this MR doesn't change.
Breaking change
Lowering the max page size of the public Project.duoWorkflowEvents field from 100 to 5 is a breaking change. It's made under the exemption for critical performance concerns. Query.duoWorkflowEvents is an experiment, so it's exempt from the deprecation process anyway. Both field descriptions name the flag and the page size with it on and off.
Performance
Measured in GDK on a seeded session with 2,000 checkpoints, one ui_chat_log message each. The query selected checkpoint and compressedCheckpoint and ran in-process with GitlabSchema.execute. Sizes are the JSON response.
| Request | Flag off | Flag on |
|---|---|---|
first: 100 |
100 rows, 22.6 s, 57.0 MB, 109 queries | 5 rows, 0.4 s, 2.9 MB, 10 queries |
first: 5 |
5 rows, 1.3 s, 2.9 MB, 13 queries | 5 rows, 0.4 s, 2.9 MB, 9 queries |
- The
first: 5row shows the batching on its own: the same five rows take about a third of the time, and the query count no longer grows with the number of rows. - Each row is still a full checkpoint, so response size per row is unchanged. The cap is what bounds the response.
- With the flag off and on, the newest rows returned identical checkpoints, both on the seeded session and on real sessions written by the Duo Workflow Service.
References
- Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/635071
- Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636470
Screenshots or screen recordings
Not applicable: this MR doesn't change any UI.
How to set up and validate locally
-
In the Rails console, seed a long session owned by a user who can run Duo Agent Platform sessions in the project. The snippet below creates 2,000 chained checkpoints and prints the session's global ID.
Seed snippet
user = User.find_by_username('root') project = Project.find_by_full_path('gitlab-org/duo-workspace') # any project where the user can use Duo Agent Platform workflow = Ai::DuoWorkflows::Workflow.create!( user: user, project: project, goal: 'Seeded long session', workflow_definition: 'software_development', agent_privileges: [Ai::DuoWorkflows::Workflow::AgentPrivileges::READ_WRITE_FILES], pre_approved_agent_privileges: [], environment: :ide, incremental_checkpoints_enabled: true ) stamps = Array.new(2_000) { Gitlab::Utils.uuid_v7 }.sort now = Time.current stamps.each_slice(1_000).with_index do |slice, s| headers, blobs = [], [] slice.each_with_index do |ts, j| i = (s * 1_000) + j headers << Ai::DuoWorkflows::CheckpointHeader.new( workflow: workflow, workflow_created_at: workflow.created_at, project: project, thread_ts: ts, parent_ts: i.zero? ? nil : stamps[i - 1], current_thread: 0, channel_keys: %w[ui_chat_log], metadata: { 'step' => i }, created_at: now, updated_at: now, checkpoint: { 'v' => 1, 'id' => ts, 'channel_versions' => { 'ui_chat_log' => i.to_s } }) message = { 'message_type' => 'agent', 'message_id' => "m-#{i}", 'content' => "message #{i} #{'x' * 200}" } blobs << Ai::DuoWorkflows::CheckpointBlob.new( workflow: workflow, workflow_created_at: workflow.created_at, project: project, thread_ts: ts, current_thread: 0, channel: 'ui_chat_log', version: i.to_s, write_type: 'json', step_action: 'conversation', data: Zlib::Deflate.deflate([message].to_json), created_at: now, updated_at: now) end Ai::DuoWorkflows::CheckpointHeader.bulk_insert!(headers) Ai::DuoWorkflows::CheckpointBlob.bulk_insert!(blobs) end workflow.to_global_id.to_s -
Signed in as that user, open GraphiQL at
http://gdk.test:3000/-/graphql-explorerand run this query with the global ID from step 1. It selectscompressedCheckpointso the server still rebuilds every row, but the browser doesn't have to render a response of over 50 MB.query($id: AiDuoWorkflowsWorkflowID!, $after: String) { duoWorkflowEvents(workflowId: $id, first: 100, after: $after) { pageInfo { hasNextPage endCursor } nodes { threadTs compressedCheckpoint } } }With the flag off, it returns 100 rows and takes around 20 seconds (22.6 s in-process in the measurement above), close to the 30-second timeout.
-
In the Rails console, enable the flag:
Feature.enable(:limit_duo_workflow_events_page_cost) -
Run the query again. It returns 5 rows in well under a second, with
hasNextPage: true. -
Pass the returned
endCursorasafter. The next 5 rows come back, so paging still reaches older checkpoints. -
To check that the content is unchanged, run the query with
first: 5andcheckpointinstead ofcompressedCheckpoint, once with the flag on and once afterFeature.disable(:limit_duo_workflow_events_page_cost). Both responses are identical.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.