Bound duoWorkflowEvents page cost behind a feature flag

What does this MR do and why?

duoWorkflowEvents rebuilds each row's checkpoint from its whole blob chain, separately for every row, and each rebuilt row carries nearly the full session state. The cost of a page grows with the number of rows times the session length. On a long session, a first: 100 request can run past the 30-second GraphQL timeout and return tens of megabytes.

Behind the limit_duo_workflow_events_page_cost flag (gitlab_com_derisk, off by default, checked against the current user), this MR:

  • Caps Query.duoWorkflowEvents and Project.duoWorkflowEvents at 5 rows per page. Larger first values are clamped, not rejected, so paging with after still reaches the whole history. The default page size stays at 1.
  • Builds checkpoint and compressedCheckpoint for a page from one blob query with a shared decode cache, the way the internal workflows API already does. Selecting both fields rebuilds each checkpoint once.

The batching is on the DuoWorkflowEvent type, so the flag also changes how checkpoint and compressedCheckpoint are built under firstCheckpoint, latestCheckpoint, and the workflowEventsUpdated subscription. Those return one event per workflow, so they still make one blob query per workflow, as before.

Who reads checkpoint:

  • The language server, which the IDE extensions and the Duo CLI use, reads it in three operations:

    • getDuoWorkflowEvents and getLatestCheckpoint in queries.ts. The IDE's agentic chat uses both, and the Duo CLI uses getLatestCheckpoint.
    • The workflowEventsUpdated subscription in workflow_events_response_channel.ts, which streams live session updates to the IDE.

    None of them pass first, so the cap doesn't change what they receive. The flag only changes how their checkpoint is built, and the result is the same.

  • The web UI only reads latestCheckpoint { duoMessages }, which this MR doesn't change.

Breaking change

Lowering the max page size of the public Project.duoWorkflowEvents field from 100 to 5 is a breaking change. It's made under the exemption for critical performance concerns. Query.duoWorkflowEvents is an experiment, so it's exempt from the deprecation process anyway. Both field descriptions name the flag and the page size with it on and off.

Performance

Measured in GDK on a seeded session with 2,000 checkpoints, one ui_chat_log message each. The query selected checkpoint and compressedCheckpoint and ran in-process with GitlabSchema.execute. Sizes are the JSON response.

Request Flag off Flag on
first: 100 100 rows, 22.6 s, 57.0 MB, 109 queries 5 rows, 0.4 s, 2.9 MB, 10 queries
first: 5 5 rows, 1.3 s, 2.9 MB, 13 queries 5 rows, 0.4 s, 2.9 MB, 9 queries
  • The first: 5 row shows the batching on its own: the same five rows take about a third of the time, and the query count no longer grows with the number of rows.
  • Each row is still a full checkpoint, so response size per row is unchanged. The cap is what bounds the response.
  • With the flag off and on, the newest rows returned identical checkpoints, both on the seeded session and on real sessions written by the Duo Workflow Service.

References

Screenshots or screen recordings

Not applicable: this MR doesn't change any UI.

How to set up and validate locally

  1. In the Rails console, seed a long session owned by a user who can run Duo Agent Platform sessions in the project. The snippet below creates 2,000 chained checkpoints and prints the session's global ID.

    Seed snippet
    user = User.find_by_username('root')
    project = Project.find_by_full_path('gitlab-org/duo-workspace') # any project where the user can use Duo Agent Platform
    workflow = Ai::DuoWorkflows::Workflow.create!(
      user: user, project: project, goal: 'Seeded long session', workflow_definition: 'software_development',
      agent_privileges: [Ai::DuoWorkflows::Workflow::AgentPrivileges::READ_WRITE_FILES],
      pre_approved_agent_privileges: [], environment: :ide, incremental_checkpoints_enabled: true
    )
    stamps = Array.new(2_000) { Gitlab::Utils.uuid_v7 }.sort
    now = Time.current
    stamps.each_slice(1_000).with_index do |slice, s|
      headers, blobs = [], []
      slice.each_with_index do |ts, j|
        i = (s * 1_000) + j
        headers << Ai::DuoWorkflows::CheckpointHeader.new(
          workflow: workflow, workflow_created_at: workflow.created_at, project: project,
          thread_ts: ts, parent_ts: i.zero? ? nil : stamps[i - 1], current_thread: 0,
          channel_keys: %w[ui_chat_log], metadata: { 'step' => i }, created_at: now, updated_at: now,
          checkpoint: { 'v' => 1, 'id' => ts, 'channel_versions' => { 'ui_chat_log' => i.to_s } })
        message = { 'message_type' => 'agent', 'message_id' => "m-#{i}", 'content' => "message #{i} #{'x' * 200}" }
        blobs << Ai::DuoWorkflows::CheckpointBlob.new(
          workflow: workflow, workflow_created_at: workflow.created_at, project: project,
          thread_ts: ts, current_thread: 0, channel: 'ui_chat_log', version: i.to_s, write_type: 'json',
          step_action: 'conversation', data: Zlib::Deflate.deflate([message].to_json), created_at: now, updated_at: now)
      end
      Ai::DuoWorkflows::CheckpointHeader.bulk_insert!(headers)
      Ai::DuoWorkflows::CheckpointBlob.bulk_insert!(blobs)
    end
    workflow.to_global_id.to_s
  2. Signed in as that user, open GraphiQL at http://gdk.test:3000/-/graphql-explorer and run this query with the global ID from step 1. It selects compressedCheckpoint so the server still rebuilds every row, but the browser doesn't have to render a response of over 50 MB.

    query($id: AiDuoWorkflowsWorkflowID!, $after: String) {
      duoWorkflowEvents(workflowId: $id, first: 100, after: $after) {
        pageInfo { hasNextPage endCursor }
        nodes { threadTs compressedCheckpoint }
      }
    }

    With the flag off, it returns 100 rows and takes around 20 seconds (22.6 s in-process in the measurement above), close to the 30-second timeout.

  3. In the Rails console, enable the flag:

    Feature.enable(:limit_duo_workflow_events_page_cost)
  4. Run the query again. It returns 5 rows in well under a second, with hasNextPage: true.

  5. Pass the returned endCursor as after. The next 5 rows come back, so paging still reaches older checkpoints.

  6. To check that the content is unchanged, run the query with first: 5 and checkpoint instead of compressedCheckpoint, once with the flag on and once after Feature.disable(:limit_duo_workflow_events_page_cost). Both responses are identical.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Charlie Linacre

Merge request reports

Loading
Loading