_Report compiled 2026-08-14 from the codebase (Rails + AI Gateway), ~35 MRs, and the issue/epic tree. Snapshot refreshed 2026-09-14; flag states verified against the feature-flag log. Design decisions are indexed below by stable `IC-` id and explained in full in the [reviewer guide](https://gitlab.com/groups/gitlab-org/-/work_items/22628#note_3692985285)._ **TLDR**: The write side is done, dual-writing globally since 2026-07-30, with a measured ~63× storage reduction (1,627 GiB → 25.7 GiB for the same week of traffic). The read side is fully merged and now live in production: `duo_workflow_read_incremental_checkpoints` at 50% of actors, `dw_read_blobs_trace` global, and `dw_read_blobs_graphql`, `dw_read_blobs_api`, `dw_read_blobs_list`, `dw_read_blobs_notifications` all at 50% of actors. The remaining path to `duo_workflow_write_incremental_only` is the tail of [&23217](https://gitlab.com/groups/gitlab-org/-/work_items/23217): merge the blob index swap [!252818](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/252818) (closes [#621934](https://gitlab.com/gitlab-org/gitlab/-/issues/621934)), take all flags global ([#604687](https://gitlab.com/gitlab-org/gitlab/-/issues/604687)), roll out write-only mode ([#607029](https://gitlab.com/gitlab-org/gitlab/-/issues/607029)), then backfill and remove the legacy table ([#611970](https://gitlab.com/gitlab-org/gitlab/-/issues/611970), [#611971](https://gitlab.com/gitlab-org/gitlab/-/issues/611971)). ## Why the change is necessary ### The trigger: an S1/P1 infradev issue [#604824 — Remove exponentially growing p_duo_workflows_checkpoints table](https://gitlab.com/gitlab-org/gitlab/-/work_items/604824) (Infrastructure, severity::1 / priority::1, SLO breached twice). The table size, in a **30-day rolling window** (partitions drop at 30 days), grew: | Date | Size | |---|---| | 2026-03-15 | 453 GB | | 2026-05-01 | 823 GB | | 2026-06-16 | 1.86 TB | | 2026-07-02 | 2.65 TB | Infra's stated risk: at this growth rate the table exhausts the main database cluster's disk within months, causing extended gitlab.com downtime. Capacity projections in [#555544](https://gitlab.com/gitlab-org/gitlab/-/work_items/555544) put plausible future load at 37.5 TB/day. ### The root cause, in code `GitLabWorkflow.aput()` (AI Gateway, `duo_workflow_service/checkpointer/gitlab_workflow.py:1455`) receives `new_versions: ChannelVersions`, which says exactly which channels changed this step, and ignored it. It serialized the **entire** checkpoint (`compress_checkpoint`, no field selection) on every LangGraph super-step. The state channels are append-only reducers (`entities/state.py:202` `_conversation_history_reducer`, `:253` `_ui_chat_log_reducer`), so state grows monotonically with step count. Per-step payload is Θ(n), total bytes over a session Θ(n²), on the wire and in Postgres. A step that only flipped `status` still re-uploaded the whole conversation. Secondary damage from the same bloat: - **Postgres misuse**: average row 361 KB (127 KB TOAST-compressed) against a ~4 KB design target. Of the legacy table's 1,627 GiB, 1,622 GiB is TOAST. - **Transport failures**: full checkpoints exceed Workhorse's gRPC message limit; the gateway has an adaptive page-size-halving retry loop down to `per_page=1` (`gitlab_workflow.py:1346-1384`). - **Read-side incident**: adding `latestCheckpoint { duoMessages }` to the session list ([!221789](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/221789)) triggered per-workflow lookups across all daily partitions and caused Dedicated incident [3702](https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/incident-management/-/issues/3702) / FCL [#105](https://gitlab.com/gitlab-com/feature-change-locks/-/work_items/105). The feature was reverted. ### Measured payoff Production shadow-write cohort, 2026-07-28 → 08-04, 90,470 workflows ([#607876](https://gitlab.com/gitlab-org/gitlab/-/work_items/607876), closed): | | Legacy | Incremental (headers + blobs) | |---|---:|---:| | Total table size | 1,627.51 GiB | 25.70 GiB (**63× smaller**) | | Per-step avg | 137 KiB | 14 KiB (−89.9%) | | Per-step p95 | 510 KiB | 51 KiB (−90.0%) | | Per-workflow p95 | 6.21 MiB | 712 KiB | Daily reduction stable at 89–92% across 41K–533K steps/day. Zero unmatched rows, zero duplicate keys in the cohort. ### Why MemorySaver-style deltas specifically - LangGraph already passes `new_versions` on every `aput`; the information is free. - LangGraph's native Postgres/SQLite blob storage was rejected: it writes a *full snapshot* per (channel, version), so at step 50 the conversation blob still holds all 50 entries. The 91.5% saving comes from domain-specific list-append deltas (`_list_delta`, `_dict_of_list_delta`), which LangGraph has no equivalent for. And `GitLabWorkflow` is a pure HTTP proxy, so Rails would have needed matching storage anyway. - Precedent in tree: `checkpoint_writes_batch` already stores interrupt writes per channel. - No data-residency change: blobs still go to the SM customer's own instance. - Trade-off accepted up front: reads become a fold over the checkpoint chain instead of one fetch, a one-time cost per resume (later bounded, see `IC-D21`). ## Design decisions Every decision carries a stable id. The rationale lives in the [reviewer guide](https://gitlab.com/groups/gitlab-org/-/work_items/22628#note_3692985285); the table below is generated from it and holds no reasoning of its own, so the two cannot drift. Cite an id in a code comment, an MR description or a review reply instead of re-explaining the decision. A decision's section is its status: 2.1-2.4 settled, 2.5 reversed, 2.6 an assumption the code depends on but nobody proved. Ids are stable and never reused. A reversed decision keeps its id and moves to 2.5, so an `IC-D48` citation that now resolves to 2.5 tells you the ground moved. | Id | Decision | § | Source | |---|---|:--:|---| | `IC-D01` | Two new tables, not a reshape of the old one | 2.1 | #605653, !244914 | | `IC-D02` | The header stores the checkpoint minus channel_values, nothing more | 2.1 | !244914 | | `IC-D03` | Blob payload is bytea, not base64 text, capped at 1 MiB by a check constraint | 2.1 | !239245 | | `IC-D04` | Both tables are range-partitioned by a dedicated workflow_created_at, daily, retained 30 days | 2.1 | #605054, !244403 | | `IC-D05` | The partition key is set from workflow.created_at on every row, including in factories | 2.1 | !244403 | | `IC-D06` | Partition-scoped has_many (the Ci::Build pattern), rather than callers remembering the bound | 2.1 | !247133 | | `IC-D07` | TTL by partition drop on the new tables | 2.1 | #605653, #611970 | | `IC-D08` | The header table is append-only: no unique index, readers take the latest | 2.1 | !244914 | | `IC-D09` | The blob dedup index includes step_action and the partition key | 2.1 | !243339 | | `IC-D10` | The blob dedup index is NULLS NOT DISTINCT | 2.1 | !239245 | | `IC-D11` | (workflow_id, thread_ts, id) on headers, dropping the standalone workflow_id index… | 2.1 | !249857 | | `IC-D12` | The JSONB columns carry no JsonSchemaValidator | 2.1 | !244914 | | `IC-D13` | Channel membership is a text[] on the header, nullable with no default | 2.1 | #613956, !250189 | | `IC-D65` | Blob reads get a dedicated (workflow_id, channel, id) index, prepared asynchronously first and swapped for the standalone workflow_id index… | 2.1 | #621934, !251705, !252818 | | `IC-D14` | Headers and blobs are built as detached instances, never through the has_many | 2.2 | !243341 | | `IC-D15` | bulk_insert! must pass unique_by: | 2.2 | !243339 | | `IC-D16` | step_action (conversation/compaction) is the append-vs-replace authority; current_thread is a grouping hint | 2.2 | !239247 | | `IC-D17` | Under write_incremental_only the service still validates the checkpoint, then skips the save | 2.2 | !247367 | | `IC-D18` | "Is this the first checkpoint?" reads the header table for incremental workflows | 2.2 | !247367 | | `IC-D19` | The goal backfill reads the __start__ blob behind the normal blob-read gate, with the payload as fallback | 2.2 | #613454 | | `IC-D56` | Channel membership is sent by the gateway as an explicit channel_keys parameter | 2.2 | !250884 | | `IC-D57` | Every incremental-only checkpoint write touches the workflow row | 2.2 | #612557, !250597 | | `IC-D66` | Development seeds write through CreateCheckpointService, never straight into the tables | 2.2 | #614099, !250816 | | `IC-D21` | The fold anchors at the last compaction and is restricted to the checkpoint's ancestor chain | 2.3 | !245415 | | `IC-D22` | Header ordering is thread_ts then id, never created_at or a bare id | 2.3 | !249621, !248330 | | `IC-D23` | Lineage lookups are scoped with for_checkpoint_ns(nil) | 2.3 | !248929 | | `IC-D24` | Dedup keys on (thread_ts, version), with per-fold step_action precedence (the channel_values fold prefers compaction; the history and… | 2.3 | !248330 | | `IC-D25` | Three folds for three questions: channel_values (agent state, one group), channel_history (display, spans compactions), channel_changes… | 2.3 | !247133, !243813, !248330 | | `IC-D26` | Blobs are the only full-history source after a compaction | 2.3 | !243813 | | `IC-D27` | Fail visible: corrupt blobs and cyclic ancestry raise | 2.3 | !247133 | | `IC-D28` | The fold uses a mutable accumulator, not value + delta | 2.3 | !247133 | | `IC-D29` | Reads are channel-scoped, and lastDuoMessage is a single LIMIT-1 blob | 2.3 | !243813 | | `IC-D31` | Consumers switch outright; never merge legacy rows with header rows | 2.3 | !249857 | | `IC-D32` | A page of checkpoints loads its blobs in one query | 2.3 | !249857 | | `IC-D54` | A header with no recorded channel membership routes the whole workflow to the legacy row | 2.3 | #617647, !250826 | | `IC-D58` | An empty channel_keys array keeps the blob path; only NULL falls back to the legacy row | 2.3 | !250826, !250884 | | `IC-D59` | The membership filter applies to state snapshots only, never to the change history | 2.3 | #613975, !251344 | | `IC-D60` | Checkpoint-presence checks batch-load with the partition key bound; never a static preload | 2.3 | !250595 | | `IC-D61` | A mid-stream compaction snapshot is folded, not dropped, with a restatement guard on the tail-append | 2.3 | !254090, !251388 | | `IC-D62` | Scalar channels only ever arrive as compaction blobs, so the changes fold records every scalar transition | 2.3 | !251410 | | `IC-D63` | The fold anchors on the lowest blob id of each compaction-group start, over the full ancestor chain | 2.3 | !251408, ai-assist!6615 | | `IC-D67` | The GraphQL presenter owns the legacy-vs-header table choice; the model carries no graphql_ methods | 2.3 | #612580, !250590 | | `IC-D68` | The membership filter runs inside the blob query, not only after the fold | 2.3 | #624938, !252495 | | `IC-D69` | The channel_values blob fetch stops at the current current_thread group, guarded by a full-chain refetch | 2.3 | #627866, !253880 | | `IC-D70` | GraphQL page fields batch-load per page with BatchLoader::GraphQL and primed header memos | 2.3 | #627867, !253884, #628051, !254155 | | `IC-D71` | The chat-log fold dedups by message_id and appends the unseen entries of a compaction snapshot | 2.3 | #628016, !254090 | | `IC-D72` | A missing intermediate ancestor is reported through error tracking, not raised | 2.3 | #627533, !253395 | | `IC-D33` | The write flag is snapshotted into incremental_checkpoints_enabled at workflow creation | 2.4 | !243345 | | `IC-D34` | duo_workflow_write_incremental_only is live-evaluated | 2.4 | #607008 | | `IC-D35` | Double gate: a master read kill switch plus a per-consumer dw_read_blobs_* flag, and every read gate also requires the workflow column | 2.4 | #604687 | | `IC-D36` | A new consumer gets its own flag rather than borrowing a similar one | 2.4 | !249857 | | `IC-D37` | No defensive fallback for "legacy row missing" | 2.4 | !249769, #612585 | | `IC-D38` | Rollout ordering is process, not code: reads global before writes stop | 2.4 | #607029 | | `IC-D39` | The gateway signal for "stop sending full checkpoints" is a server capability | 2.4 | !250083 | | `IC-D40` | by_thread_ts has no fallback and no Rails-side flag re-check | 2.4 | #606987 | | `IC-D41` | Blobs are JSON, not LangGraph msgpack | 2.4 | ai-assist!6138 | | `IC-D42` | Every blob group is self-contained: a compaction re-seeds all channels as full snapshots | 2.4 | ai-assist!6179 | | `IC-D43` | The gateway blobs every channel; the scalar allowlist goes away | 2.4 | ai-assist!6526 | | `IC-D44` | Nothing may depend on compressed_checkpoint, or on the legacy p_duo_workflows_checkpoints table | 2.4 | #613454, !249521, !249745 | | `IC-D55` | Legacy-read enforcement is the RuboCop cop now, and a table rename at the end, not a query analyzer | 2.4 | !250618, #616856 | | `IC-D64` | The legacy-read cop and the runtime warning share one exclusion list | 2.4 | !249745, #612557 | | `IC-A03` | "Headers are a superset of the legacy table" is enforced, no longer a convention | 2.4 | !250850, #614034 | | `IC-D45` | The legacy-row backfill lands only after the write flag is default-enabled, and the legacy read code goes one required stop later | 2.4 | #611970, #611971 | | `IC-D73` | Every dw_read_blobs_* flag carries log_state_changes: true | 2.4 | !251973 | | `IC-D74` | The gateway raises when a checkpoint POST fails and advances the delta baseline only after a 2xx | 2.4 | #627533, ai-assist!6747 | | `IC-D75` | The gateway trims ui_chat_log at every compaction under write_incremental_only | 2.4 | #628017, ai-assist!6818 | | `IC-D46` | current_thread_started_at marker on the checkpoint as a >= partition bound | 2.5 | — | | `IC-D47` | Deterministic created_at decoded from thread_ts (a UUIDv6/v7 decoder in the model) | 2.5 | — | | `IC-D48` | Walking the parent_ts chain across the whole workflow to reconstruct | 2.5 | — | | `IC-D49` | A unique dedup index on the header table | 2.5 | — | | `IC-D50` | A second partial unique index for namespace-level blob rows (WHERE project_id IS NULL) | 2.5 | !239245 | | `IC-D51` | No backfill of the legacy table | 2.5 | #611970 | | `IC-D52` | A Rails-side reconstruction of non-blobbed scalars | 2.5 | — | | `IC-D53` | Piping the write-only signal to the gateway as a pushed feature flag | 2.5 | — | | `IC-D20` | No new API parameter when the data is already server-side | 2.5 | !250884 | | `IC-D30` | Reads reuse the dedup index rather than adding read-only indexes | 2.5 | !247133, !251705 | | `IC-A09` | The accepted per-checkpoint blob queries on the GraphQL paths | 2.5 | !253884 | | `IC-A01` | The kill-switch coverage of by_thread_ts is indirect | 2.6 | — | | `IC-A02` | "Latest checkpoint" is max(thread_ts) with an id tie-break, not the leaf of the parent_ts chain | 2.6 | #607648 | | `IC-A04` | "Rollout ordering is enforced by process." | 2.6 | — | | `IC-A05` | Header channel_keys is authoritative only if the gateway blobs every channel | 2.6 | ai-assist!6526, !250884, ai-assist!6575, #613975, !251344 | | `IC-A06` | "Workflows are short-lived", so a session alive past the 30-day retention cannot write | 2.6 | — | | `IC-A07` | "Fail visible" is right for agent state, and questionable for display | 2.6 | — | | `IC-A08` | The fold's cost bound assumes every unbounded channel eventually compacts | 2.6 | ai-assist!6818 | | `IC-A10` | Append-only headers put the burden on every reader | 2.6 | !249621 | ### Open thread: which channels get blobbed Reviewers keep re-deriving this sequence. Lists and dicts-of-lists only, then `status` carved out as an always-blobbed scalar. QA on [!247134](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/247134) then found LangGraph's ephemeral `branch:to:*` routing channels missing under write-incremental-only, so a resumed graph sees zero armed tasks and terminates silently. Gateway [!6526](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/6526) therefore drops the allowlist and blobs every channel (`IC-D43`). Blobs still cannot express channel *deletion*: `channel_keys` membership on the header (`IC-D13`) plus a fold filter ([#613975](https://gitlab.com/gitlab-org/gitlab/-/issues/613975)) closes that half, and the membership list is authoritative only once !6526 ships (`IC-A05`). ## Feature flags & rollout Every flag is `type: wip`, `default_enabled: false`, `group::agent execution`. Fill the last two columns as each flag is turned on. (Table refreshed 2026-09-10.) | Flag | Gates | MR | Rollout issue | Enabled for `gitlab-org` | Enabled globally | |------|-------|----|---------------|:------------------------:|:----------------:| | `duo_workflow_incremental_checkpoints` | Global switch: Shadow-writes blobs + slim header alongside the full checkpoint (dual write) | on master (phase 1) | gitlab-org/gitlab#604684 | :rocket: | :rocket: | | `duo_workflow_write_incremental_only` | Stops shadow-writing the full checkpoint — headers + blobs only | gitlab-org/gitlab!247367 | gitlab-org/gitlab#607029 | | | | `duo_workflow_read_incremental_checkpoints` | Master off-switch for all blob reads; every read consumer also requires this | gitlab-org/gitlab!247133 | gitlab-org/gitlab#604687 | :rocket: | :rocket: (09-10) | | `dw_read_blobs_api` | Internal checkpoint API / gateway (`by_thread_ts`) | gitlab-org/gitlab!247134 | gitlab-org/gitlab#606987 | :rocket: | :rocket: (09-10) | | `dw_read_blobs_trace` | Trace download (`trace.jsonl`) | gitlab-org/gitlab!247135 | gitlab-org/gitlab#606988 | :rocket: | :rocket: (08-19) | | `dw_read_blobs_graphql` | GraphQL checkpoint fields (`duoMessages`, `lastDuoMessage`, `executionStatus`, raw `checkpoint`, `latestCheckpoint`/`firstCheckpoint` lookup) | gitlab-org/gitlab!243813, gitlab-org/gitlab!247391, gitlab-org/gitlab!247392, gitlab-org/gitlab!249802 | gitlab-org/gitlab#606989 | :rocket: | :rocket: (09-10) | | `dw_read_blobs_notifications` | Messaging / notifications (callback worker, progress reader, result email) | gitlab-org/gitlab!247369 | gitlab-org/gitlab#606990 | :rocket: | 50% of actors (09-02) | | `dw_read_blobs_list` | Internal checkpoint list endpoint (`checkpoints_reversed`, pre-18.8 fallback) | gitlab-org/gitlab!249857 | gitlab-org/gitlab#614021 | :rocket: | 50% of actors (09-02) | ### Rollout procedure Order is dependency-driven: reads must be fully on blobs **before** the write side stops emitting the full checkpoint. 1. **Write blobs (dual write)** — `duo_workflow_incremental_checkpoints`. Blobs + slim headers accumulate next to the full checkpoint. Must reach global before read rollout so every workflow has blobs to read from. *(Done: global since 2026-07-30.)* 2. **Read kill switch** — enable `duo_workflow_read_incremental_checkpoints`. On its own this changes nothing (each consumer also needs its own flag); it is the master off-switch for the whole read path. 3. **Read consumers, one at a time** — for each `dw_read_blobs_*`, enable for `gitlab-org`, validate dashboards, then go global. A consumer reconstructs from blobs only when the kill switch **and** its own flag are on, so consumers roll out and roll back independently. 4. **Stop the dual write** — only after **all** read consumers are global and stable (blobs are the sole read source), enable `duo_workflow_write_incremental_only` for `gitlab-org`, then global. This drops the legacy full-checkpoint write. Gated by every child of gitlab-org&23217. Per flag — `gitlab-org` first, then global: ``` /chatops gitlab run feature set --group=gitlab-org <flag> true # gitlab-org /chatops gitlab run feature set <flag> true # global ``` Rollback a single consumer: `/chatops gitlab run feature set <flag> false`. Disable the entire read path at once: `/chatops gitlab run feature set duo_workflow_read_incremental_checkpoints false`. ## Current snapshot (2026-09-14) Flag states verified against the feature-flag log, MR and issue states against the API. No flag changed since 2026-09-10. ### Flags (gprd) | Flag | Scope | % of actors | |---|---|---| | `duo_workflow_incremental_checkpoints` | **global** (07-30) | — | | `duo_workflow_read_incremental_checkpoints` | **global** (09-10) — kill switch | — | | `dw_read_blobs_trace` | **global** (08-19) | — | | `dw_read_blobs_api` | **global** (09-10) | — | | `dw_read_blobs_graphql` | **global** (09-10) | — | | `dw_read_blobs_notifications` | gitlab-org (08-19) | **50%** (09-02) | | `dw_read_blobs_list` | gitlab-org (08-19) | **50%** (09-02) | | `duo_workflow_write_incremental_only` | off — 2 open prerequisites in [&23217](https://gitlab.com/groups/gitlab-org/-/work_items/23217): [#626571](https://gitlab.com/gitlab-org/gitlab/-/issues/626571) (blob index swap) and [#627867](https://gitlab.com/gitlab-org/gitlab/-/issues/627867) (language-server switch to `lastDuoMessage`) | — | Since 2026-09-10 the gateway resume path, the trace download and every GraphQL checkpoint field read blobs for every incremental workflow. Notifications and the internal list endpoint stay at 50% of actors; with the kill switch global, their joint coverage is exactly their own percentage. **Incident 09-01**: [production#22837](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/22837) (S4, duo-workflow-svc gRPC error rate). The kill switch was disabled at 05:34 UTC as a suspect; the error rate did not change, and the cause was traced to the status-update path (a reconnect on a finished workflow classified as RETRY, rejected by Rails as an invalid transition). Restored at 06:51 UTC at 25% of actors + gitlab-org. Blob reads are not implicated. ### Merged since 09-02 - ai-assist [!6747](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/6747) (09-04) — raise when a checkpoint POST fails; the delta baseline advances only after a 2xx (`IC-D74`, gateway half of #627533) - [!253395](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253395) (09-07) — report truncated checkpoint ancestry via error tracking (`IC-D72`, Rails half; closes [#627533](https://gitlab.com/gitlab-org/gitlab/-/issues/627533)) - [!253884](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253884) (09-08) — batch `latestCheckpoint` and `duoMessages` across GraphQL pages (`IC-D70`; ~131 queries per 20-row page down to 4 per page; backend part of #627867) - [!253880](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253880) (09-08) — bound the `channel_values` blob fetch at the `current_thread` group, with a full-chain refetch guard (`IC-D69`; closes [#627866](https://gitlab.com/gitlab-org/gitlab/-/issues/627866) on 09-14) - [!254155](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/254155) (09-08) — batch `firstCheckpoint` across GraphQL pages (`IC-D70`; closes [#628051](https://gitlab.com/gitlab-org/gitlab/-/issues/628051) on 09-14) - [!254090](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/254090) (09-08) — `message_id` dedup in the chat-log fold, unseen compaction snapshot entries kept (`IC-D71`; closes [#628016](https://gitlab.com/gitlab-org/gitlab/-/issues/628016)) - ai-assist [!6818](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/6818) (09-09) — trim `ui_chat_log` at compaction under `write_incremental_only` (`IC-D75`; closes [#628017](https://gitlab.com/gitlab-org/gitlab/-/issues/628017)) Nothing merged between 09-10 and 09-14. ### Open MRs | MR | Issue | State | |---|---|---| | [!252818](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/252818) swap the blob `workflow_id` index for `(workflow_id, channel, id)` | [#626571](https://gitlab.com/gitlab-org/gitlab/-/issues/626571) | database review; pipeline green. The reviewer's Database Lab check on 09-10 found 29 of 58 partitions with a valid index (20260811 → 20260908), 20 queued in `postgres_async_indexes` with 0 attempts, and 9 partitions created after the prepare migration (20260929 → 20261007) not queued. Partition 20260909 alone is 2.5M rows / 8.6 GB, so merging now would build indexes synchronously in the post-deploy migration. The reviewer asked for four things before merge: drain the 20 queued entries (0 attempts needs a look), enqueue or accept synchronous builds for the 9 unqueued partitions, re-run the partition check right before merging, and confirm old-index scans on Grafana. Author pinged 09-11; no reply yet | ### Still open | Issue | State | |---|---| | [#626571](https://gitlab.com/gitlab-org/gitlab/-/issues/626571) blob index swap | &23217 prerequisite; !252818 blocked on the async index build (see above) | | [#627867](https://gitlab.com/gitlab-org/gitlab/-/issues/627867) IDE session list N+1 | now a &23217 prerequisite; backend batching merged, remaining item is the language server switching to `lastDuoMessage` | | [#607029](https://gitlab.com/gitlab-org/gitlab/-/issues/607029), [#611970](https://gitlab.com/gitlab-org/gitlab/-/issues/611970), [#611971](https://gitlab.com/gitlab-org/gitlab/-/issues/611971) | rollout → BBM → legacy-read removal tail | | Flag rollout issues #606987–#606990, #614021, #604686, #604687 | chatops + cleanup | ### Next steps 1. Unblock [!252818](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/252818): check why the 20 queued async indexes show 0 attempts, enqueue the 9 unqueued partitions (or accept synchronous builds for them), then re-run the partition check and land it. That closes #626571. 2. Land the language-server switch to `lastDuoMessage` for [#627867](https://gitlab.com/gitlab-org/gitlab/-/issues/627867). With #626571 that empties &23217 and unblocks `duo_workflow_write_incremental_only`. 3. Take `dw_read_blobs_notifications` and `dw_read_blobs_list` from 50% to global; every other read flag is already global ([#604687](https://gitlab.com/gitlab-org/gitlab/-/issues/604687)). 4. Then [#607029](https://gitlab.com/gitlab-org/gitlab/-/issues/607029) gitlab-org → percentage → global; [#611970](https://gitlab.com/gitlab-org/gitlab/-/issues/611970) → [#611971](https://gitlab.com/gitlab-org/gitlab/-/issues/611971), which includes the legacy-table rename. **Critical path**: !252818 + #627867 → notifications/list flags global → #607029 → #611970 → #611971.