Use array form for single-workflow checkpoint blob thread_ts reads

Eduardo Bonetrequested to merge
631438-pass-thread-ts-chains-as-an-array-in-the-rema into master

What does this MR do and why?

Planning for long checkpoint blob reads drops from up to 98 ms to about 0.2 ms, because three single-workflow reads now send the thread_ts chain as one array (CheckpointBlob.for_thread_ts) instead of a literal IN list. On busy partitions, long chains with a channel filter (history_blobs_for, checkpoint_statuses) pay about 1.5 to 2 ms more execution in return, while the last-message preview keeps its 4-buffer read. The batched page query keeps the literal list, because there the array becomes a row filter that gets slower as the page grows.

Design notes

This MR is stacked on !258660 (merged) and targets its branch. That MR added Ai::DuoWorkflows::CheckpointBlob.for_thread_ts, which builds thread_ts = ANY(ARRAY(SELECT unnest(...))) instead of a literal IN list. Review and merge that MR first; this one will show only its own commit once it merges.

Why the array form: a literal IN list costs planning time per value, roughly list length times the size of the partition's thread_ts most-common-values (MCV) list. The MCV list has 838 to 1,000 entries on low-volume partitions (1.2M rows or fewer, mostly weekend days plus the current day's partition while it's small) and 4 to 258 entries on the others. At around 1,100 values the literal list plans in 80 to 98 ms on a low-volume partition and about 2 ms on a busy one. The array form plans in about 0.2 ms on every partition tested.

What changed in ee/app/models/ai/duo_workflows/workflow.rb:

Method Serves Filter
full_trace_channel_values -> accumulated_blobs_for trace.jsonl download channel IN ('ui_chat_log', 'plan', 'status') by default, none with full=true; longest chain (full ancestor chain across every current_thread group)
checkpoint_statuses duoWorkflowBranches GraphQL field, via workflow_branches / build_branches channel = 'status'
history_blobs_for channel_message_history, build_branches per-channel
latest_channel_message last-message preview (latestCheckpoint { lastDuoMessage }), ORDER BY id DESC LIMIT 1 per-channel, no project_id

Workflow.history_blobs_for_workflows (the batched GraphQL message history for a page of workflows, via WorkflowCheckpointEventPresenter.batch_duo_messages) stays on the literal list, with a comment explaining why. The history_blobs_for comment also drops the index name, because the array form can change which index serves the query, and now notes it also receives branch subtrees from build_branches, not only an ancestor chain.

Database Lab numbers, planning ms / execution ms / shared buffers, literal vs array:

  • Workflow 8711521 (2026-09-27, MCV 1,000, 1,139 values): history_blobs_for 86.6 / 0.9 / 696 vs 0.2 / 3.8 / 790; checkpoint_statuses 79.1 / 0.5 / 500 vs 0.2 / 3.8 / 598; default trace 84.5 / 1.5 / 1194 vs 0.3 / 2.4 / 1108; full=true trace 98.2 / 5.8 / 1476 vs 0.2 / 4.6 / 1476.
  • Workflow 8746344 (2026-09-28, MCV 52, 1,184 values): history_blobs_for 2.0 / 0.6 / 689 vs 0.2 / 4.3 / 781; checkpoint_statuses 2.0 / 0.4 / 412 vs 0.2 / 4.1 / 508; default trace 2.1 / 1.7 / 1099 vs 0.2 / 2.4 / 1149; full=true trace 2.0 / 5.5 / 1710 vs 0.2 / 4.5 / 1710.
  • Namespace-level workflow 8789107 (project_id IS NULL, 2026-09-29, MCV 22, 711 values): history_blobs_for 0.9 / 0.4 / 364 vs 0.2 / 2.5 / 420. The array form still keeps project_id IS NULL in the index condition.
  • Short lists (5 and 50 values): identical plans and buffers in both forms. At 50 values the literal list still planned in 4.3 to 8.5 ms on the 2026-09-27 partition, versus 0.2 ms for the array form.

latest_channel_message (the last-message preview) also drops project_id, on top of the array form. With project_id, the planner picks idx_duo_wf_checkpoint_blobs_dedup and reads and sorts every matching blob to return one row; without it, it scans idx_duo_wf_checkpoint_blobs_on_workflow_channel_id backwards and stops at the newest match. Array form, with vs without project_id: 8746344 (1,184 values) 0.17 / 1.99 / 781 vs 0.23 / 0.32 / 4; 8711521 (1,139 values) 0.21 / 1.96 / 790 vs 0.16 / 0.23 / 4; 8789107 namespace-level (711 values) 0.20 / 1.82 / 420 vs 0.20 / 0.14 / 4. The literal list on master also reads 4 buffers, but plans in up to 88.6 ms on a low-volume partition.

Trade-off to know about: on busy partitions, a long chain with a channel filter still loses the no-sort plan for history_blobs_for and checkpoint_statuses, costing about 1.5 to 2 ms more execution and about 15% more buffers (buffers are the stable measure; cold-cache Database Lab timings run much higher than warm ones for these two reads). The default three-channel trace pays about 0.7 ms more execution and up to 5% more buffers on busy partitions, and the full=true trace is no worse anywhere. In exchange, planning drops by 80 to 98 ms on low-volume partitions. latest_channel_message has its own trade-off: dropping project_id keeps the fast plan for the latest checkpoint, but for an older checkpoint the backward scan has to pass newer off-chain blobs and check each against the array (118 values: 0.2 ms / 85 buffers becomes 1.7 ms / 624 buffers; 592 values: 1.0 ms / 386 buffers becomes 4.0 ms / 356 buffers). The list preview, latestCheckpoint { lastDuoMessage }, passes the latest checkpoint, which is the fast case.

Why the page query (history_blobs_for_workflows) keeps the literal list: it also filters on workflow_id IN (...) and project_id IN (...). With those in place, the planner keeps the (workflow_id, channel, id) index and applies the array as a row filter. Postgres only hashes a constant array, so the filter checks each row against every value, and cost grows with rows times chain length. Page of 20 workflows, 1 partition, 2,010 values: literal 2.5 / 1.4 / 1020 vs array 0.6 / 18.1 / 1020 (up to 22.8 ms execution in another run). Pages spanning 5 partitions, about 300 values: literal 47.5 to 51.6 / 0.5 vs array 3.4 / 0.8. So the page query still pays about 50 ms of planning on multi-partition pages. IN (SELECT unnest(...)) looked better on long pages (2,010 values: 0.9 / 4.0), but MR 258660 found it picks nested-loop plans that read the whole channel at 25 to 400 values, so it needs its own size sweep before it's worth adopting here. The evidence is posted on the issue.

New specs check that #channel_message_history, #workflow_branches and #full_trace_channel_values send thread_ts = ANY(ARRAY(SELECT unnest(...))), and that #latest_channel_message leaves project_id out of its query, in ee/spec/models/ai/duo_workflows/workflow_spec.rb. The scope's SQL shape and escaping stay covered separately in ee/spec/models/ai/duo_workflows/checkpoint_blob_spec.rb (added in MR 258660). Spec runs, all passing: workflow_spec.rb plus channel_values_reconstructor_spec.rb (704 examples), the presenter and workflow_branches request specs (66 examples), and the workflows request spec (563 examples).

References

Screenshots or screen recordings

No UI change.

How to set up and validate locally

  1. Open a Rails console on GDK (gdk rails console) and find a workflow with incremental checkpoints: header = Ai::DuoWorkflows::CheckpointHeader.order(:id).last then workflow = header.workflow.
  2. Turn on SQL logging: ActiveRecord::Base.logger = Logger.new($stdout).
  3. Run workflow.latest_channel_message(header, 'ui_chat_log'). Expect: the query uses thread_ts = ANY(ARRAY(SELECT unnest('{...}'::text[]))), ORDER BY ... DESC LIMIT 1, no project_id filter, and the result matches master.
  4. Run workflow.full_trace_channel_values(channels: %w[ui_chat_log plan status]) (the default trace.jsonl channels). Expect: the blob query uses the ANY form with "channel" IN (...), and the result hash matches master.
  5. Run workflow.workflow_branches(workflow.chat_log_thread_ts.max). Expect: a channel = 'status' blob query in the ANY form. The result is an empty array unless a turn was retried.
  6. Alternative without a real session: bin/rspec ee/spec/models/ai/duo_workflows/workflow_spec.rb.
Query plans (Database Lab)

Source: Database Lab gitlab-production-main, 2026-09-30 session (clone data as of 2026-09-30 04:24 UTC). Probes ran as DO blocks that build the query server-side with EXECUTE and raise the plan as an error (Joe hides rows). Each query ran twice; plans below are the warm run. Array literals are elided as {...}. The chain is every thread_ts of the workflow's headers.

Workflows: 8711521 (project 84861201, 2026-09-27 partition, thread_ts MCV list of 1,000 entries, 1,139 values), 8746344 (project 278964, 2026-09-28 partition, MCV list of 52 entries, 1,184 values), and 8789107 (namespace-level, project_id IS NULL, 2026-09-29 partition, MCV list of 22 entries, 711 values).

SQL on this branch. On master, every query has "p_duo_workflows_checkpoint_blobs"."project_id" = ... and "p_duo_workflows_checkpoint_blobs"."thread_ts" IN (...) in place of the ANY clause.

-- Workflow#history_blobs_for (also Workflow#checkpoint_statuses, with channel = 'status')
SELECT "p_duo_workflows_checkpoint_blobs".* FROM "p_duo_workflows_checkpoint_blobs" WHERE "p_duo_workflows_checkpoint_blobs"."workflow_id" = 8746344 AND "p_duo_workflows_checkpoint_blobs"."workflow_created_at" = '2026-09-28 18:06:20.188548' AND "p_duo_workflows_checkpoint_blobs"."project_id" = 278964 AND "p_duo_workflows_checkpoint_blobs"."channel" = 'ui_chat_log' AND ("p_duo_workflows_checkpoint_blobs".thread_ts = ANY(ARRAY(SELECT unnest('{...}'::text[])))) ORDER BY "p_duo_workflows_checkpoint_blobs"."id" ASC

-- Workflow#latest_channel_message (no project_id)
SELECT "p_duo_workflows_checkpoint_blobs".* FROM "p_duo_workflows_checkpoint_blobs" WHERE "p_duo_workflows_checkpoint_blobs"."workflow_id" = 8746344 AND "p_duo_workflows_checkpoint_blobs"."workflow_created_at" = '2026-09-28 18:06:20.188548' AND "p_duo_workflows_checkpoint_blobs"."channel" = 'ui_chat_log' AND ("p_duo_workflows_checkpoint_blobs".thread_ts = ANY(ARRAY(SELECT unnest('{...}'::text[])))) ORDER BY "p_duo_workflows_checkpoint_blobs"."id" DESC LIMIT 1

-- Workflow#full_trace_channel_values via #accumulated_blobs_for, default trace.jsonl (TRACE_SAFE_CHANNELS)
SELECT "p_duo_workflows_checkpoint_blobs".* FROM "p_duo_workflows_checkpoint_blobs" WHERE "p_duo_workflows_checkpoint_blobs"."workflow_id" = 8746344 AND "p_duo_workflows_checkpoint_blobs"."workflow_created_at" = '2026-09-28 18:06:20.188548' AND "p_duo_workflows_checkpoint_blobs"."project_id" = 278964 AND ("p_duo_workflows_checkpoint_blobs".thread_ts = ANY(ARRAY(SELECT unnest('{...}'::text[])))) AND "p_duo_workflows_checkpoint_blobs"."channel" IN ('ui_chat_log', 'plan', 'status') ORDER BY "p_duo_workflows_checkpoint_blobs"."id" ASC

-- Workflow#full_trace_channel_values via #accumulated_blobs_for, trace.jsonl with full=true
SELECT "p_duo_workflows_checkpoint_blobs".* FROM "p_duo_workflows_checkpoint_blobs" WHERE "p_duo_workflows_checkpoint_blobs"."workflow_id" = 8746344 AND "p_duo_workflows_checkpoint_blobs"."workflow_created_at" = '2026-09-28 18:06:20.188548' AND "p_duo_workflows_checkpoint_blobs"."project_id" = 278964 AND ("p_duo_workflows_checkpoint_blobs".thread_ts = ANY(ARRAY(SELECT unnest('{...}'::text[])))) ORDER BY "p_duo_workflows_checkpoint_blobs"."id" ASC

Workflow#history_blobs_for

Before, workflow 8746344 (busy partition):

Index Scan using index_c8cdf89081 on p_duo_workflows_checkpoint_blobs_20260928 p_duo_workflows_checkpoint_blobs  (cost=3.39..570.62 rows=1 width=403) (actual time=0.054..0.515 rows=697 loops=1)
  Index Cond: ((workflow_id = 8746344) AND (channel = 'ui_chat_log'::text))
  Filter: ((workflow_created_at = '2026-09-28 18:06:20.188548+00'::timestamp with time zone) AND (project_id = 278964) AND (thread_ts = ANY ('{...}'::text[])))
  Buffers: shared hit=689
Planning Time: 1.919 ms
Execution Time: 0.559 ms

After, workflow 8746344 (busy partition):

Sort  (cost=28.18..28.19 rows=1 width=403) (actual time=2.115..2.162 rows=697 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 537kB
  Buffers: shared hit=781
  InitPlan 1
    ->  ProjectSet  (cost=0.00..5.94 rows=1184 width=32) (actual time=0.001..0.076 rows=1184 loops=1)
          ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.001 rows=1 loops=1)
  ->  Index Scan using p_duo_workflows_checkpoint_b_project_id_workflow_id_threa_idx34 on p_duo_workflows_checkpoint_blobs_20260928 p_duo_workflows_checkpoint_blobs  (cost=0.56..22.24 rows=1 width=403) (actual time=0.534..1.897 rows=697 loops=1)
        Index Cond: ((project_id = 278964) AND (workflow_id = 8746344) AND (thread_ts = ANY ((InitPlan 1).col1)) AND (channel = 'ui_chat_log'::text) AND (workflow_created_at = '2026-09-28 18:06:20.188548+00'::timestamp with time zone))
        Buffers: shared hit=781
Planning Time: 0.224 ms
Execution Time: 2.221 ms

Before, workflow 8711521 (low-volume partition):

Index Scan using index_22d319300b on p_duo_workflows_checkpoint_blobs_20260927 p_duo_workflows_checkpoint_blobs  (cost=3.27..476.49 rows=1 width=368) (actual time=0.057..0.571 rows=712 loops=1)
  Index Cond: ((workflow_id = 8711521) AND (channel = 'ui_chat_log'::text))
  Filter: ((workflow_created_at = '2026-09-27 13:05:42.680577+00'::timestamp with time zone) AND (project_id = 84861201) AND (thread_ts = ANY ('{...}'::text[])))
  Buffers: shared hit=696
Planning Time: 83.507 ms
Execution Time: 0.621 ms

After, workflow 8711521 (low-volume partition):

Sort  (cost=27.91..27.91 rows=1 width=368) (actual time=1.952..1.988 rows=712 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 535kB
  Buffers: shared hit=790
  InitPlan 1
    ->  ProjectSet  (cost=0.00..5.71 rows=1139 width=32) (actual time=0.001..0.068 rows=1139 loops=1)
          ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.000 rows=1 loops=1)
  ->  Index Scan using p_duo_workflows_checkpoint_b_project_id_workflow_id_threa_idx33 on p_duo_workflows_checkpoint_blobs_20260927 p_duo_workflows_checkpoint_blobs  (cost=0.55..22.19 rows=1 width=368) (actual time=0.469..1.734 rows=712 loops=1)
        Index Cond: ((project_id = 84861201) AND (workflow_id = 8711521) AND (thread_ts = ANY ((InitPlan 1).col1)) AND (channel = 'ui_chat_log'::text) AND (workflow_created_at = '2026-09-27 13:05:42.680577+00'::timestamp with time zone))
        Buffers: shared hit=790
Planning Time: 0.183 ms
Execution Time: 2.035 ms

Workflow#latest_channel_message

Before, workflow 8746344 (busy partition):

Limit  (cost=3.39..570.62 rows=1 width=403) (actual time=0.060..0.060 rows=1 loops=1)
  Buffers: shared hit=4
  ->  Index Scan Backward using index_c8cdf89081 on p_duo_workflows_checkpoint_blobs_20260928 p_duo_workflows_checkpoint_blobs  (cost=3.39..570.62 rows=1 width=403) (actual time=0.059..0.059 rows=1 loops=1)
        Index Cond: ((workflow_id = 8746344) AND (channel = 'ui_chat_log'::text))
        Filter: ((workflow_created_at = '2026-09-28 18:06:20.188548+00'::timestamp with time zone) AND (project_id = 278964) AND (thread_ts = ANY ('{...}'::text[])))
        Buffers: shared hit=4
Planning Time: 1.868 ms
Execution Time: 0.074 ms

After, workflow 8746344 (busy partition):

Limit  (cost=6.37..576.69 rows=1 width=403) (actual time=0.291..0.292 rows=1 loops=1)
  Buffers: shared hit=4
  InitPlan 1
    ->  ProjectSet  (cost=0.00..5.94 rows=1184 width=32) (actual time=0.002..0.113 rows=1184 loops=1)
          ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.001 rows=1 loops=1)
  ->  Index Scan Backward using index_c8cdf89081 on p_duo_workflows_checkpoint_blobs_20260928 p_duo_workflows_checkpoint_blobs  (cost=0.43..570.76 rows=1 width=403) (actual time=0.291..0.291 rows=1 loops=1)
        Index Cond: ((workflow_id = 8746344) AND (channel = 'ui_chat_log'::text))
        Filter: ((workflow_created_at = '2026-09-28 18:06:20.188548+00'::timestamp with time zone) AND (thread_ts = ANY ((InitPlan 1).col1)))
        Buffers: shared hit=4
Planning Time: 0.229 ms
Execution Time: 0.315 ms

Before, workflow 8711521 (low-volume partition):

Limit  (cost=3.27..476.49 rows=1 width=368) (actual time=0.060..0.061 rows=1 loops=1)
  Buffers: shared hit=4
  ->  Index Scan Backward using index_22d319300b on p_duo_workflows_checkpoint_blobs_20260927 p_duo_workflows_checkpoint_blobs  (cost=3.27..476.49 rows=1 width=368) (actual time=0.059..0.060 rows=1 loops=1)
        Index Cond: ((workflow_id = 8711521) AND (channel = 'ui_chat_log'::text))
        Filter: ((workflow_created_at = '2026-09-27 13:05:42.680577+00'::timestamp with time zone) AND (project_id = 84861201) AND (thread_ts = ANY ('{...}'::text[])))
        Buffers: shared hit=4
Planning Time: 76.470 ms
Execution Time: 0.080 ms

After, workflow 8711521 (low-volume partition):

Limit  (cost=6.14..481.97 rows=1 width=368) (actual time=0.213..0.213 rows=1 loops=1)
  Buffers: shared hit=4
  InitPlan 1
    ->  ProjectSet  (cost=0.00..5.71 rows=1139 width=32) (actual time=0.001..0.099 rows=1139 loops=1)
          ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.000 rows=1 loops=1)
  ->  Index Scan Backward using index_22d319300b on p_duo_workflows_checkpoint_blobs_20260927 p_duo_workflows_checkpoint_blobs  (cost=0.42..476.26 rows=1 width=368) (actual time=0.212..0.212 rows=1 loops=1)
        Index Cond: ((workflow_id = 8711521) AND (channel = 'ui_chat_log'::text))
        Filter: ((workflow_created_at = '2026-09-27 13:05:42.680577+00'::timestamp with time zone) AND (thread_ts = ANY ((InitPlan 1).col1)))
        Buffers: shared hit=4
Planning Time: 0.164 ms
Execution Time: 0.229 ms

After, namespace-level workflow 8789107:

Limit  (cost=4.00..303.46 rows=1 width=408) (actual time=0.126..0.126 rows=1 loops=1)
  Buffers: shared hit=4
  InitPlan 1
    ->  ProjectSet  (cost=0.00..3.57 rows=711 width=32) (actual time=0.001..0.050 rows=711 loops=1)
          ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.000 rows=1 loops=1)
  ->  Index Scan Backward using index_fa9959bae8 on p_duo_workflows_checkpoint_blobs_20260929 p_duo_workflows_checkpoint_blobs  (cost=0.43..299.89 rows=1 width=408) (actual time=0.125..0.125 rows=1 loops=1)
        Index Cond: ((workflow_id = 8789107) AND (channel = 'ui_chat_log'::text))
        Filter: ((workflow_created_at = '2026-09-29 15:20:22.260057+00'::timestamp with time zone) AND (thread_ts = ANY ((InitPlan 1).col1)))
        Buffers: shared hit=4
Planning Time: 0.200 ms
Execution Time: 0.143 ms

Workflow#checkpoint_statuses

Before, workflow 8746344 (busy partition):

Index Scan using index_c8cdf89081 on p_duo_workflows_checkpoint_blobs_20260928 p_duo_workflows_checkpoint_blobs  (cost=3.39..249.42 rows=1 width=403) (actual time=0.054..0.335 rows=411 loops=1)
  Index Cond: ((workflow_id = 8746344) AND (channel = 'status'::text))
  Filter: ((workflow_created_at = '2026-09-28 18:06:20.188548+00'::timestamp with time zone) AND (project_id = 278964) AND (thread_ts = ANY ('{...}'::text[])))
  Buffers: shared hit=412
Planning Time: 1.827 ms
Execution Time: 0.365 ms

After, workflow 8746344 (busy partition):

Sort  (cost=28.18..28.19 rows=1 width=403) (actual time=1.766..1.793 rows=411 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 89kB
  Buffers: shared hit=508
  InitPlan 1
    ->  ProjectSet  (cost=0.00..5.94 rows=1184 width=32) (actual time=0.001..0.084 rows=1184 loops=1)
          ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.000 rows=1 loops=1)
  ->  Index Scan using p_duo_workflows_checkpoint_b_project_id_workflow_id_threa_idx34 on p_duo_workflows_checkpoint_blobs_20260928 p_duo_workflows_checkpoint_blobs  (cost=0.56..22.24 rows=1 width=403) (actual time=0.532..1.694 rows=411 loops=1)
        Index Cond: ((project_id = 278964) AND (workflow_id = 8746344) AND (thread_ts = ANY ((InitPlan 1).col1)) AND (channel = 'status'::text) AND (workflow_created_at = '2026-09-28 18:06:20.188548+00'::timestamp with time zone))
        Buffers: shared hit=508
Planning Time: 0.205 ms
Execution Time: 1.837 ms

Before, workflow 8711521 (low-volume partition):

Index Scan using index_22d319300b on p_duo_workflows_checkpoint_blobs_20260927 p_duo_workflows_checkpoint_blobs  (cost=3.27..277.98 rows=1 width=368) (actual time=0.060..0.491 rows=506 loops=1)
  Index Cond: ((workflow_id = 8711521) AND (channel = 'status'::text))
  Filter: ((workflow_created_at = '2026-09-27 13:05:42.680577+00'::timestamp with time zone) AND (project_id = 84861201) AND (thread_ts = ANY ('{...}'::text[])))
  Buffers: shared hit=500
Planning Time: 79.512 ms
Execution Time: 0.534 ms

After, workflow 8711521 (low-volume partition):

Sort  (cost=27.91..27.91 rows=1 width=368) (actual time=1.868..1.897 rows=506 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 107kB
  Buffers: shared hit=598
  InitPlan 1
    ->  ProjectSet  (cost=0.00..5.71 rows=1139 width=32) (actual time=0.002..0.072 rows=1139 loops=1)
          ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.001 rows=1 loops=1)
  ->  Index Scan using p_duo_workflows_checkpoint_b_project_id_workflow_id_threa_idx33 on p_duo_workflows_checkpoint_blobs_20260927 p_duo_workflows_checkpoint_blobs  (cost=0.55..22.19 rows=1 width=368) (actual time=0.532..1.765 rows=506 loops=1)
        Index Cond: ((project_id = 84861201) AND (workflow_id = 8711521) AND (thread_ts = ANY ((InitPlan 1).col1)) AND (channel = 'status'::text) AND (workflow_created_at = '2026-09-27 13:05:42.680577+00'::timestamp with time zone))
        Buffers: shared hit=598
Planning Time: 0.212 ms
Execution Time: 1.953 ms

Default trace.jsonl (three channels)

Before, workflow 8746344 (busy partition):

Sort  (cost=863.68..863.69 rows=1 width=403) (actual time=1.566..1.630 rows=1108 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 626kB
  Buffers: shared hit=1099
  ->  Index Scan using index_c8cdf89081 on p_duo_workflows_checkpoint_blobs_20260928 p_duo_workflows_checkpoint_blobs  (cost=3.39..863.67 rows=1 width=403) (actual time=0.085..1.217 rows=1108 loops=1)
        Index Cond: ((workflow_id = 8746344) AND (channel = ANY ('{...}'::text[])))
        Filter: ((workflow_created_at = '2026-09-28 18:06:20.188548+00'::timestamp with time zone) AND (project_id = 278964) AND (thread_ts = ANY ('{...}'::text[])))
        Buffers: shared hit=1099
Planning Time: 2.113 ms
Execution Time: 1.700 ms

After, workflow 8746344 (busy partition):

Sort  (cost=69.63..69.64 rows=1 width=403) (actual time=2.913..2.977 rows=1108 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 626kB
  Buffers: shared hit=1149
  InitPlan 1
    ->  ProjectSet  (cost=0.00..5.94 rows=1184 width=32) (actual time=0.001..0.076 rows=1184 loops=1)
          ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.000 rows=1 loops=1)
  ->  Index Scan using p_duo_workflows_checkpoint_b_project_id_workflow_id_threa_idx34 on p_duo_workflows_checkpoint_blobs_20260928 p_duo_workflows_checkpoint_blobs  (cost=0.56..63.68 rows=1 width=403) (actual time=0.556..2.586 rows=1108 loops=1)
        Index Cond: ((project_id = 278964) AND (workflow_id = 8746344) AND (thread_ts = ANY ((InitPlan 1).col1)) AND (channel = ANY ('{...}'::text[])) AND (workflow_created_at = '2026-09-28 18:06:20.188548+00'::timestamp with time zone))
        Buffers: shared hit=1149
Planning Time: 0.211 ms
Execution Time: 3.046 ms

Before, workflow 8711521 (low-volume partition):

Sort  (cost=821.50..821.50 rows=1 width=368) (actual time=1.728..1.822 rows=1218 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 642kB
  Buffers: shared hit=1194
  ->  Index Scan using index_22d319300b on p_duo_workflows_checkpoint_blobs_20260927 p_duo_workflows_checkpoint_blobs  (cost=3.27..821.49 rows=1 width=368) (actual time=0.071..1.340 rows=1218 loops=1)
        Index Cond: ((workflow_id = 8711521) AND (channel = ANY ('{...}'::text[])))
        Filter: ((workflow_created_at = '2026-09-27 13:05:42.680577+00'::timestamp with time zone) AND (project_id = 84861201) AND (thread_ts = ANY ('{...}'::text[])))
        Buffers: shared hit=1194
Planning Time: 84.517 ms
Execution Time: 1.917 ms

After, workflow 8711521 (low-volume partition):

Sort  (cost=69.26..69.26 rows=1 width=368) (actual time=3.291..3.382 rows=1218 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 642kB
  Buffers: shared hit=1108
  InitPlan 1
    ->  ProjectSet  (cost=0.00..5.71 rows=1139 width=32) (actual time=0.003..0.105 rows=1139 loops=1)
          ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.001 rows=1 loops=1)
  ->  Index Scan using p_duo_workflows_checkpoint_b_project_id_workflow_id_threa_idx33 on p_duo_workflows_checkpoint_blobs_20260927 p_duo_workflows_checkpoint_blobs  (cost=0.55..63.53 rows=1 width=368) (actual time=0.579..2.885 rows=1218 loops=1)
        Index Cond: ((project_id = 84861201) AND (workflow_id = 8711521) AND (thread_ts = ANY ((InitPlan 1).col1)) AND (channel = ANY ('{...}'::text[])) AND (workflow_created_at = '2026-09-27 13:05:42.680577+00'::timestamp with time zone))
        Buffers: shared hit=1108
Planning Time: 0.349 ms
Execution Time: 3.467 ms

trace.jsonl with full=true (every channel)

Before, workflow 8746344 (busy partition):

Sort  (cost=2437.40..2437.41 rows=1 width=403) (actual time=3.085..3.302 rows=3503 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 1556kB
  Buffers: shared hit=1710
  ->  Index Scan using p_duo_workflows_checkpoint_b_project_id_workflow_id_threa_idx34 on p_duo_workflows_checkpoint_blobs_20260928 p_duo_workflows_checkpoint_blobs  (cost=0.56..2437.39 rows=1 width=403) (actual time=0.387..2.159 rows=3503 loops=1)
        Index Cond: ((project_id = 278964) AND (workflow_id = 8746344) AND (thread_ts = ANY ('{...}'::text[])) AND (workflow_created_at = '2026-09-28 18:06:20.188548+00'::timestamp with time zone))
        Buffers: shared hit=1710
Planning Time: 2.095 ms
Execution Time: 3.466 ms

After, workflow 8746344 (busy partition):

Sort  (cost=28.16..28.16 rows=1 width=403) (actual time=3.108..3.317 rows=3503 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 1556kB
  Buffers: shared hit=1710
  InitPlan 1
    ->  ProjectSet  (cost=0.00..5.94 rows=1184 width=32) (actual time=0.001..0.081 rows=1184 loops=1)
          ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.000 rows=1 loops=1)
  ->  Index Scan using p_duo_workflows_checkpoint_b_project_id_workflow_id_threa_idx34 on p_duo_workflows_checkpoint_blobs_20260928 p_duo_workflows_checkpoint_blobs  (cost=0.56..22.21 rows=1 width=403) (actual time=0.525..2.195 rows=3503 loops=1)
        Index Cond: ((project_id = 278964) AND (workflow_id = 8746344) AND (thread_ts = ANY ((InitPlan 1).col1)) AND (workflow_created_at = '2026-09-28 18:06:20.188548+00'::timestamp with time zone))
        Buffers: shared hit=1710
Planning Time: 0.167 ms
Execution Time: 3.484 ms

Before, workflow 8711521 (low-volume partition):

Sort  (cost=2298.06..2298.06 rows=1 width=368) (actual time=4.257..4.703 rows=3528 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 1977kB
  Buffers: shared hit=1476
  ->  Index Scan using p_duo_workflows_checkpoint_b_project_id_workflow_id_threa_idx33 on p_duo_workflows_checkpoint_blobs_20260927 p_duo_workflows_checkpoint_blobs  (cost=0.55..2298.05 rows=1 width=368) (actual time=0.617..3.089 rows=3528 loops=1)
        Index Cond: ((project_id = 84861201) AND (workflow_id = 8711521) AND (thread_ts = ANY ('{...}'::text[])) AND (workflow_created_at = '2026-09-27 13:05:42.680577+00'::timestamp with time zone))
        Buffers: shared hit=1476
Planning Time: 79.921 ms
Execution Time: 4.934 ms

After, workflow 8711521 (low-volume partition):

Sort  (cost=27.88..27.89 rows=1 width=368) (actual time=3.068..3.286 rows=3528 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 1977kB
  Buffers: shared hit=1476
  InitPlan 1
    ->  ProjectSet  (cost=0.00..5.71 rows=1139 width=32) (actual time=0.001..0.070 rows=1139 loops=1)
          ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.000 rows=1 loops=1)
  ->  Index Scan using p_duo_workflows_checkpoint_b_project_id_workflow_id_threa_idx33 on p_duo_workflows_checkpoint_blobs_20260927 p_duo_workflows_checkpoint_blobs  (cost=0.55..22.16 rows=1 width=368) (actual time=0.494..2.167 rows=3528 loops=1)
        Index Cond: ((project_id = 84861201) AND (workflow_id = 8711521) AND (thread_ts = ANY ((InitPlan 1).col1)) AND (workflow_created_at = '2026-09-27 13:05:42.680577+00'::timestamp with time zone))
        Buffers: shared hit=1476
Planning Time: 0.172 ms
Execution Time: 3.455 ms

Namespace-level workflow 8789107, Workflow#history_blobs_for

Before:

Index Scan using index_fa9959bae8 on p_duo_workflows_checkpoint_blobs_20260929 p_duo_workflows_checkpoint_blobs  (cost=2.21..299.08 rows=1 width=408) (actual time=0.052..0.281 rows=372 loops=1)
  Index Cond: ((workflow_id = 8789107) AND (channel = 'ui_chat_log'::text))
  Filter: ((project_id IS NULL) AND (workflow_created_at = '2026-09-29 15:20:22.260057+00'::timestamp with time zone) AND (thread_ts = ANY ('{...}'::text[])))
  Buffers: shared hit=364
Planning Time: 0.954 ms
Execution Time: 0.310 ms

After:

Sort  (cost=25.82..25.82 rows=1 width=408) (actual time=1.196..1.214 rows=372 loops=1)
  Sort Key: p_duo_workflows_checkpoint_blobs.id
  Sort Method: quicksort  Memory: 299kB
  Buffers: shared hit=420
  InitPlan 1
    ->  ProjectSet  (cost=0.00..3.57 rows=711 width=32) (actual time=0.001..0.045 rows=711 loops=1)
          ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.000..0.000 rows=1 loops=1)
  ->  Index Scan using p_duo_workflows_checkpoint_b_project_id_workflow_id_threa_idx35 on p_duo_workflows_checkpoint_blobs_20260929 p_duo_workflows_checkpoint_blobs  (cost=0.56..22.24 rows=1 width=408) (actual time=0.312..1.092 rows=372 loops=1)
        Index Cond: ((project_id IS NULL) AND (workflow_id = 8789107) AND (thread_ts = ANY ((InitPlan 1).col1)) AND (channel = 'ui_chat_log'::text) AND (workflow_created_at = '2026-09-29 15:20:22.260057+00'::timestamp with time zone))
        Buffers: shared hit=420
Planning Time: 0.177 ms
Execution Time: 1.249 ms

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Eduardo Bonet

Merge request reports