<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/engineering/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=627995) </details> <!--IssueSummary end--> ## Summary Three queries behind `GET /api/:version/runners/:id/jobs` regress on PostgreSQL 18. This is the largest **unaddressed** source of user-facing PG18 failures found so far, and the highest-priority item under #605358. In a 15-minute production sample across four PG18 replicas during the GPRD CI mock upgrade, this endpoint accounted for **44 statements cancelled at the application's 15s command timeout and 664 wasted backend-seconds** — second only to the redundant-pipeline cancellation family, whose interim fix has already merged. One of the three queries fails **37% of the time**. Analysis: [dbo-issue-tracker#755](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/755). Parent: #605358. | queryid | kills | executions | % killed | p50 | profile | |---|---|---|---|---|---| | `-4994932943211833455` | 35 | 95 | **37%** | 3.7s | **CPU / not-waiting** | | `5903599151245115449` | 8 | 24 | 33% | 1.9s | IO-bound | | `720887595379886518` | 1 | 93 | 1% | 1.2s | CPU / not-waiting | Earlier clone analysis (2026-07) measured this endpoint flipping in **100% of sampled parameter sets** at **×3.3–5.2 buffers** — the only query in that pass to flip unconditionally, classified as an intrinsic regression. `5903599151245115449` was separately measured at **10.9 s of block-read time per call**, the worst per-call figure in its capture. This has been visibly bad for two months. Note the p50 of 3.7s: the endpoint is slow even when it does not fail. > [!NOTE] > **Reclassified 2026-09-10.** The `Problem` section below replaces the original > appendrel diagnosis. The captured statements show this is the PG18 btree > skip-scan problem (problem 2 in the DBO analysis), not the appendrel problem > (problem 1). The parent link to #605358, "rewrite filtered joins on partitioned > CI tables to LATERAL", may therefore no longer be the right home for this > issue. Read the caveat on the evidence before acting on any of it. > > `720887595379886518` in the table above is the endpoint's count probe, not a > page query. ## Problem ### What the failing statements actually are All three killed statements were captured from `pg_stat_activity`. They share one shape, with no status filter, no `system_id` join, and no partition predicate: ```sql SELECT <all 47 p_ci_builds columns> FROM p_ci_builds WHERE type = 'Ci::Build' AND runner_id = $1 AND project_id IN (<ci_project_mirrors UNION subquery>) ORDER BY id DESC LIMIT 101 [OFFSET 100] ``` - `-4994932943211833455` is page 2 (`LIMIT 101 OFFSET 100`), runner `51700412`, namespace `59715498`. - `5903599151245115449` is page 1 (`LIMIT 101`), runner `40406427`, namespace `5369312`. - `720887595379886518` is not a page query. It is the count probe: `SELECT COUNT(*) FROM (SELECT 1 AS one FROM p_ci_builds WHERE ... LIMIT 10001) subquery_for_count`. It comes from `Gitlab::Pagination::OffsetPagination#exceeeds_count?` calling `total_count_with_limit`, and it runs on every default-path request before the page query. Source for all three: [dbo-issue-tracker#755 note_3773004668](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/755#note_3773004668). ### Consequences for the original problem statement Two of the five numbered items in the original problem statement are not supported by any observed failure. The claimed correlated status filter (`by_status` -> `with_statuses`) appears in none of the three captured statements. The claimed second partitioned join on the `system_id` path (`belonging_to_runner_manager` -> `p_ci_runner_machine_builds`) appears in none of them either. The `system_id` path does not show up in the capture at all. The `fbc0fe9a2e` appendrel mechanism needs a filter that correlates with join membership on an equi-join against a unique key. Neither precondition holds here. The `project_id` semi-join is a hash semi-join on a non-unique column. Three items do hold: - No partition pruning anywhere in the finder or the endpoint. - The `project_id` semi-join against `ci_project_mirrors`. - `LIMIT`/`OFFSET` pagination plus the count probe, because `without_count: true` is only set when `system_id` is present. `LIMIT 101` rather than `LIMIT 100` means `exceeeds_count?` already returned true for these runners. They have more than 10,000 jobs, so the endpoint had already fallen back to `without_count`. The count probe still ran first. ### The actual mechanism: PG18 btree skip scan, not appendrel The DBO analysis tracks two separate PG18 problems ([dbo-issue-tracker#755 note_3542918669](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/755#note_3542918669)): - Problem 1, appendrel, commit [`fbc0fe9a2e`](https://git.postgresql.org/cgit/postgresql.git/commit/?id=fbc0fe9a2e). - Problem 2, the new btree skip scan, which lets the planner use an index whose leading column has no predicate. This endpoint is problem 2. The PG18 plan ([dbo-issue-tracker#755 note_3548755153](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/755#note_3548755153)) is: ``` Limit (cost=8844186.95..8844197.12 rows=4065 width=547) -> Sort (cost=8844186.95..8844288.57 rows=40648 width=547) Sort Key: p_ci_builds.id DESC -> Hash Join (cost=8729324.87..8841075.17 rows=40648 width=547) Hash Cond: (p_ci_builds.project_id = ci_project_mirrors.project_id) -> Append (cost=0.71..111085.40 rows=81295 width=547) -> Index Scan using index_ci_builds_on_status_and_type_and_runner_id on ci_builds Index Cond: ((type = $1) AND (runner_id = $2)) -> Index Scan using ci_builds_101_status_type_runner_id_idx on ci_builds_101 Index Cond: ((type = $1) AND (runner_id = $2)) ... 12 more index scans, same shape, on ci_builds_102 .. ci_builds_113 ... -> Seq Scan on ci_builds_114 (the empty newest partition) -> Hash (the ci_project_mirrors UNION subquery) ``` PG18 scans `(status, type, runner_id)` with equality on `type` and `runner_id` and nothing on the leading `status` column. That index does not return rows in `id` order, so the plan needs a full `Sort` of roughly 40,000 rows at 547 bytes each before the `LIMIT 101`. The `LIMIT` early exit is gone. `p_ci_builds_runner_id_id_idx` on `(runner_id, id DESC)` already exists on every partition. It would let a Merge Append feed the hash join in `id DESC` order and stop after 101 matching rows. PG17 has no skip scan, so that index is the expected PG17 choice, but the PG17 plan has not been captured yet and this remains to be confirmed. This explains the otherwise odd CPU profile: 431 backend-seconds not-waiting against 97 in IO for `-4994932943211833455`. The cost is a large in-memory sort of wide rows, not extra IO. It is consistent with the independently observed PG18 temp-file and sort-spill regression ([dbo-issue-tracker#755 note_3720172595](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/755#note_3720172595)). ### Caveat on the evidence > [!WARNING] > The July plan above is `EXPLAIN (buffers, verbose, settings, generic_plan)` run against the normalized SQL, not real literals. Two consequences follow. First, the index choice in the `Append` is still trustworthy. `type = $1 AND runner_id = $2` is plain equality on real columns, and skip-scan eligibility for an index whose leading column has no predicate is structural rather than literal-dependent. Second, the `ci_project_mirrors` subtree in that plan is not trustworthy, and its 8.7M cost should be ignored. Under generic parameters, `WHERE 1=0` becomes `WHERE $3 = $4`, which the planner cannot fold away, and `traversal_ids[1]` becomes `traversal_ids[$5]`, which cannot use `index_ci_namespace_mirrors_on_traversal_ids_unnest`. Both distortions are artefacts of generic planning and disappear with real literals. So the skip-scan diagnosis must be confirmed with `EXPLAIN (ANALYZE, BUFFERS)` at the real failing literals before any code is written. PG18 has no GUC to switch skip scan off and adds no distinct plan node. The way to detect it is the `Index Searches: N` line that PG18 added to `EXPLAIN (ANALYZE)`. A value above 1 on an index whose leading column has no predicate is the skip scan. ## Proposal Ordered by cost and certainty. Each step is independently shippable and measurable. ### Step 0 — confirm the mechanism Run `EXPLAIN (ANALYZE, BUFFERS, VERBOSE, SETTINGS)` for all three statements at the real failing literals above, on a PG18 clone and a PG17 clone. Report the `Index Searches: N` line for every index scan, whether a `Sort` sits between the `Limit` and the join, and `Append` versus `Merge Append`. To test the hypothesis without a GUC, hide the suspect indexes from the planner inside a transaction (`UPDATE pg_index SET indisvalid = false ... ; EXPLAIN ...; ROLLBACK`) on a thin clone. If the plan then uses `(runner_id, id DESC)` with a `Merge Append` and no `Sort`, the diagnosis is settled. If Step 0 does not confirm skip scan, stop and re-plan. Steps 1 and 2 hold either way; Step 3 depends on it. ### Step 1 — drop the count probe Make `without_count: true` unconditional for this endpoint in `lib/api/ci/runners.rb`, behind a `gitlab_com_derisk` flag. This deletes `720887595379886518` by construction rather than by plan tuning, and halves the query count on the default path. Behaviour change: `X-Total` and `X-Total-Pages` disappear for runners with 10,000 jobs or fewer. Runners above that already lose them, which is why the captured statements are `LIMIT 101`. The `system_id` path already behaves this way, and `doc/api/runners.md` already documents keyset, which also omits those headers, as the preferred method. Needs a docs update. ### Step 2 — narrow the select The entity is `Entities::Ci::JobBasicWithProject`. It reaches 19 of the 47 columns: - exposed: `id status name ref tag coverage allow_failure created_at started_at finished_at erased_at queued_at failure_reason` - structural: `partition_id` (`query_constraints`, and the `in_partition` association scopes), `type` (STI), `project_id` (`web_url`, project entity), `commit_id` (aliased `pipeline_id`; drives `pipeline`, `commit`, `sha`), `stage_id` (`stage`), `user_id` (`user`) `duration` and `queued_duration` are computed in `commit_status.rb`, not columns. That cuts row width from 547 bytes to roughly 120. If the `Sort` survives, this still cuts sort volume and temp-file spill by about 4.5x, on PG17 as well as PG18, which is why it ships independently of Step 3. ### Step 3 — restore the ordered index scan Chosen after Step 0's evidence, cheapest first. 1. **Keyset by default for this path.** A cursor adds `id < $n`, which `(runner_id, id DESC)` can use as a second index condition and the skip-scan index cannot. That should make the ordered index unambiguously cheaper and remove deep-offset cost. `Ci::Build.supported_keyset_orderings` already returns `{ id: [:asc, :desc] }` and `CursorBasedKeyset` already supports it. Page 1 has no cursor, so it needs its own measurement. A keyset-shaped query elsewhere in the same capture (`-4944403423927718015`) still timed out, so this is not a guaranteed cure. 2. **Reconsider `p_ci_builds_status_type_runner_id_idx`.** It is the index PG18 latches onto. Reordering it, or dropping it if `p_ci_builds_runner_id_id_idx` and the partial `p_ci_builds_runner_id_idx` cover its real consumers, removes the temptation. Needs a consumer audit and `~database` review, so treat it as a fallback. 3. **`by_permission` / the `ci_project_mirrors` UNION.** Only if Step 0's `BUFFERS` output shows real time in the hash build. The 8.7M cost in the July plan is a generic-parameter artefact and is not evidence. ### Not doing: partition pruning Adding `in_partition(Ci::Partition.recent_ids)`, as `app/graphql/resolvers/ci/runner_job_count_resolver.rb` does, is deliberately out of scope. `Ci::Partition.recent_ids` returns the three most recent `current` or `active` partitions, so jobs in older partitions would stop appearing. That is acceptable for a job *count* and a stronger contract to break for a job *list*. It also would not have prevented these failures: a partition-pruned query in the same capture (`-5214854549358047953`, `partition_id = 101`) was killed at 15s anyway. ### Not doing: LATERAL, and the `system_id` path The original Step 3 proposed a LATERAL rewrite of the `system_id` path, following `app/services/ci/timed_out_builds/drop_running_service.rb`. Dropped, for two reasons. The `system_id` path appears in no captured failure, and it already sets `without_count: true`. And the pattern does not transfer: the `drop_running_service.rb` rewrite fixes an equi-join on a unique key against a small tracking table, whereas the failing statements here hash-join on `project_id`, which is not unique in `p_ci_builds`. ## Verification - Use the **failing parameter sets** from the two SQL files attached to [dbo-issue-tracker#755 note_3773253281](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/755#note_3773253281), not synthesised or median ones. Two queries elsewhere in this family were cleared on median-parameter testing in July and later killed real requests. - `EXPLAIN (ANALYZE, BUFFERS)` on **both** PG17 and PG18 clones. The fix ships before the upgrade, so it must not regress PG17. - Real literals only. A `generic_plan` on the normalized SQL misreports the `ci_project_mirrors` subtree, as described in the caveat above. ## Acceptance criteria - [ ] Mechanism confirmed or refuted by Step 0, with the `Index Searches` counts recorded on both versions. - [ ] All three queryids addressed: `-4994932943211833455`, `5903599151245115449`, `720887595379886518`. - [ ] Verified at the failing parameter sets, not median ones, and with real literals rather than `generic_plan`. - [ ] `EXPLAIN (ANALYZE, BUFFERS)` on PG18: no timeout at the failing parameters; no `Sort` between the `Limit` and the join; `Merge Append` over `(runner_id, id DESC)`; `Limit` actual rows about 101; the ×3.3–5.2 buffer regression gone. - [ ] `EXPLAIN (ANALYZE, BUFFERS)` on PG17: no regression. - [ ] **The not-waiting component explained.** If the fix removes IO but leaves ~431 backend-seconds of not-waiting, it has not solved the actual failure. Measure this specifically for `-4994932943211833455`. - [ ] Whether `-136894955912632268` and `720887595379886518` are the same statement under PG17 and PG18 queryids is settled on dbo-issue-tracker#755. The July reading of `-136894955912632268` as "neutral, ~65 ms" cannot stand alongside a count probe on this endpoint being killed at 15s. - [ ] p50 improves from 3.7s. - [ ] Response ordering unchanged and request specs pass. The loss of `X-Total` and `X-Total-Pages` from Step 1 is documented and behind a flag. ## Verification update (2026-09-23) This section supersedes the parts of the existing mechanism, proposal, and acceptance criteria that expect hiding the skip-scan indexes to produce `Merge Append` and remove the `Sort`. PostgreSQL 18 btree skip scans are confirmed by `Index Searches: 19-22` on the `(status, type, runner_id)` indexes. Hiding all 14 status/type/runner indexes restores the `(runner_id, id)` access paths with `Index Searches: 1`. However, both the baseline and hidden-index plans retain the same overall shape: ```text Append -> Hash Join -> top-N Sort -> Limit ``` The total buffer counts barely change: - `-4994932943211833455`: `15,730 -> 14,589` - `5903599151245115449`: `59,674 -> 59,011` The lower execution times in the hidden-index runs are confounded by cache warmth and cannot establish causality. The experiment confirms that the indexes cause PostgreSQL 18 to choose skip scans, but it does not prove that this access-path change causes the performance regression. Index DDL is therefore not justified by the current evidence. ### Exact example query for each queryid Query ID `-4994932943211833455`: ```sql SELECT "p_ci_builds"."status", "p_ci_builds"."finished_at", "p_ci_builds"."created_at", "p_ci_builds"."updated_at", "p_ci_builds"."started_at", "p_ci_builds"."coverage", "p_ci_builds"."name", "p_ci_builds"."allow_failure", "p_ci_builds"."stage_idx", "p_ci_builds"."tag", "p_ci_builds"."ref", "p_ci_builds"."type", "p_ci_builds"."target_url", "p_ci_builds"."description", "p_ci_builds"."erased_at", "p_ci_builds"."artifacts_expire_at", "p_ci_builds"."environment", "p_ci_builds"."when", "p_ci_builds"."queued_at", "p_ci_builds"."lock_version", "p_ci_builds"."coverage_regex", "p_ci_builds"."retried", "p_ci_builds"."protected", "p_ci_builds"."failure_reason", "p_ci_builds"."scheduled_at", "p_ci_builds"."token_encrypted", "p_ci_builds"."resource_group_id", "p_ci_builds"."waiting_for_resource_at", "p_ci_builds"."processed", "p_ci_builds"."scheduling_type", "p_ci_builds"."id", "p_ci_builds"."stage_id", "p_ci_builds"."partition_id", "p_ci_builds"."auto_canceled_by_partition_id", "p_ci_builds"."auto_canceled_by_id", "p_ci_builds"."commit_id", "p_ci_builds"."erased_by_id", "p_ci_builds"."project_id", "p_ci_builds"."runner_id", "p_ci_builds"."upstream_pipeline_id", "p_ci_builds"."user_id", "p_ci_builds"."upstream_pipeline_partition_id", "p_ci_builds"."scoped_user_id", "p_ci_builds"."timeout", "p_ci_builds"."timeout_source", "p_ci_builds"."exit_code", "p_ci_builds"."debug_trace_enabled" FROM "p_ci_builds" WHERE "p_ci_builds"."type" = 'Ci::Build' AND "p_ci_builds"."runner_id" = 51700412 AND "p_ci_builds"."project_id" IN ( SELECT "ci_project_mirrors"."project_id" FROM ( (SELECT "ci_project_mirrors".* FROM "ci_project_mirrors" WHERE 1 = 0) UNION (SELECT "ci_project_mirrors".* FROM "ci_project_mirrors" WHERE "ci_project_mirrors"."namespace_id" IN ( SELECT "ci_namespace_mirrors"."namespace_id" FROM ( (SELECT "ci_namespace_mirrors".* FROM "ci_namespace_mirrors" WHERE traversal_ids[1] IN (59715498)) ) "ci_namespace_mirrors" )) ) "ci_project_mirrors" ) ORDER BY "p_ci_builds"."id" DESC LIMIT 101 OFFSET 100; ``` Query ID `5903599151245115449`: ```sql SELECT "p_ci_builds"."status", "p_ci_builds"."finished_at", "p_ci_builds"."created_at", "p_ci_builds"."updated_at", "p_ci_builds"."started_at", "p_ci_builds"."coverage", "p_ci_builds"."name", "p_ci_builds"."allow_failure", "p_ci_builds"."stage_idx", "p_ci_builds"."tag", "p_ci_builds"."ref", "p_ci_builds"."type", "p_ci_builds"."target_url", "p_ci_builds"."description", "p_ci_builds"."erased_at", "p_ci_builds"."artifacts_expire_at", "p_ci_builds"."environment", "p_ci_builds"."when", "p_ci_builds"."queued_at", "p_ci_builds"."lock_version", "p_ci_builds"."coverage_regex", "p_ci_builds"."retried", "p_ci_builds"."protected", "p_ci_builds"."failure_reason", "p_ci_builds"."scheduled_at", "p_ci_builds"."token_encrypted", "p_ci_builds"."resource_group_id", "p_ci_builds"."waiting_for_resource_at", "p_ci_builds"."processed", "p_ci_builds"."scheduling_type", "p_ci_builds"."id", "p_ci_builds"."stage_id", "p_ci_builds"."partition_id", "p_ci_builds"."auto_canceled_by_partition_id", "p_ci_builds"."auto_canceled_by_id", "p_ci_builds"."commit_id", "p_ci_builds"."erased_by_id", "p_ci_builds"."project_id", "p_ci_builds"."runner_id", "p_ci_builds"."upstream_pipeline_id", "p_ci_builds"."user_id", "p_ci_builds"."upstream_pipeline_partition_id", "p_ci_builds"."scoped_user_id", "p_ci_builds"."timeout", "p_ci_builds"."timeout_source", "p_ci_builds"."exit_code", "p_ci_builds"."debug_trace_enabled" FROM "p_ci_builds" WHERE "p_ci_builds"."type" = 'Ci::Build' AND "p_ci_builds"."runner_id" = 40406427 AND "p_ci_builds"."project_id" IN ( SELECT "ci_project_mirrors"."project_id" FROM ( (SELECT "ci_project_mirrors".* FROM "ci_project_mirrors" WHERE 1 = 0) UNION (SELECT "ci_project_mirrors".* FROM "ci_project_mirrors" WHERE "ci_project_mirrors"."namespace_id" IN ( SELECT "ci_namespace_mirrors"."namespace_id" FROM ( (SELECT "ci_namespace_mirrors".* FROM "ci_namespace_mirrors" WHERE traversal_ids[1] IN (5369312)) ) "ci_namespace_mirrors" )) ) "ci_project_mirrors" ) ORDER BY "p_ci_builds"."id" DESC LIMIT 101; ``` Query ID `720887595379886518`: ```sql SELECT COUNT(*) FROM ( SELECT 1 AS one FROM "p_ci_builds" WHERE "p_ci_builds"."type" = 'Ci::Build' AND "p_ci_builds"."runner_id" = 45808894 AND "p_ci_builds"."project_id" IN ( SELECT "ci_project_mirrors"."project_id" FROM ( (SELECT "ci_project_mirrors".* FROM "ci_project_mirrors" WHERE 1 = 0) UNION (SELECT "ci_project_mirrors".* FROM "ci_project_mirrors" WHERE "ci_project_mirrors"."namespace_id" IN ( SELECT "ci_namespace_mirrors"."namespace_id" FROM ( (SELECT "ci_namespace_mirrors".* FROM "ci_namespace_mirrors" WHERE traversal_ids[1] IN (59715498)) ) "ci_namespace_mirrors" )) ) "ci_project_mirrors" ) LIMIT 10001 ) "subquery_for_count"; ``` ### Large-table constraint The [large-table limitations](https://docs.gitlab.com/development/database/large_tables_limitations/) apply. A table larger than 50 GB cannot receive another index without an approved schema-change exception. Replacing an existing index is an allowed exception category, but that does not grant automatic approval. Before dropping, reordering, or replacing a status/type/runner index, the investigation must establish: - A repository-wide audit of every consumer of the index. - GitLab.com index-usage and query telemetry. - Coverage of self-managed installations and infrequent jobs that may not appear in short GitLab.com samples. - Evidence that alternative indexes efficiently cover every identified access pattern. - Database team review and approval of the schema-change exception. Any approved migration of indexes across the partitioned table must use safe asynchronous and concurrent mechanisms. The implementation should not be prescribed until the required evidence and approval determine whether index DDL is appropriate. ### Revised implementation plan **A. Remove the count probe as the first independent candidate.** Retain removal of `720887595379886518` behind a `gitlab_com_derisk` flag. This is independently useful because it removes one query from the request path rather than trying to tune its plan. **B. Narrow the selected columns as a separate candidate.** Retain the explicit narrow projection as a schema-free change that can be measured independently. It may reduce row width, memory use, and sorting work even if the overall plan shape does not change. **C. Investigate query shapes that reduce rows before the hash join and sort.** Evaluate changes to the permission semi-join and pagination using the real literals above on PostgreSQL 17 and PostgreSQL 18. Each candidate must show that it reduces the rows or work reaching the hash join and top-N sort. Keyset pagination must be measured for both page 1 and later pages. A result that improves only pages with a cursor is insufficient. **D. Treat index DDL as a last-resort, gated investigation.** Index DDL may be investigated only after the consumer and usage evidence is complete and the required large-table exception is approved. The old Step 3 option to replace or drop the status/type/runner index is no longer an implementation option at this stage. ### Revised acceptance criteria - [ ] All three query IDs are addressed: `-4994932943211833455`, `5903599151245115449`, and `720887595379886518`. - [ ] Every query-plan comparison uses the real failing literals on both PostgreSQL 17 and PostgreSQL 18. - [ ] Verification uses row-returning `EXPLAIN (ANALYZE, BUFFERS)` plans rather than estimates or generic plans. - [ ] None of the three queries reaches the 15-second application timeout at the failing parameter sets. - [ ] Buffers or other structural work are materially reduced. `Merge Append` and removal of the `Sort` are not mandatory plan-shape requirements. - [ ] Endpoint p50 improves from the recorded 3.7 seconds. - [ ] The production not-waiting component is explained and shown to improve. - [ ] Response contents, ordering, pagination behavior, and header behavior are tested and documented. - [ ] Any index DDL is supported by consumer and usage evidence and has an approved large-table schema-change exception.