## Summary `TestSumDistinctBlobSizes_PlanIsIndexOnlyOnOneShadowPartition`, in `internal/datastore/statistics_footprint_integration_test.go`, fails intermittently in `test:integration` on every PostgreSQL version the job matrixes over, on branches that touch nothing near it. The test pins an `Index Only Scan` on a partition of `pk_blob_storage_blobs_by_namespace`. The planner instead reaches the narrow `blob_storage_blobs_by_namespace_pNN_namespace_id_size_idx` through a Bitmap Index Scan, puts a Bitmap Heap Scan above it, and joins with a nested loop or a hash join, so the test's `scanned` variable stays empty and the `require.NotEmptyf` at line 333 fails. `test:integration` is a required job, so every failure latches its pipeline: a concluded-failed pipeline does not recompute from a job retry, and an armed auto-merge waits on it until a fresh MR pipeline runs. On 2026-08-26 the test failed six jobs across five pipelines, dropped a merge train car, and held a second merge request until someone rolled a fresh pipeline by hand. ## Occurrences on 2026-08-26 | Job | PG | Pipeline | Source | UTC | | --- | --- | --- | --- | --- | | [16121656946](https://gitlab.com/gitlab-org/ops/artifact-registry/-/jobs/16121656946) | 16 | [2792844390](https://gitlab.com/gitlab-org/ops/artifact-registry/-/pipelines/2792844390) | step-11 MR, merge ref | 14:11 | | [16121656947](https://gitlab.com/gitlab-org/ops/artifact-registry/-/jobs/16121656947) | 17 | [2792844390](https://gitlab.com/gitlab-org/ops/artifact-registry/-/pipelines/2792844390) | step-11 MR, merge ref | 14:11 | | [16124986305](https://gitlab.com/gitlab-org/ops/artifact-registry/-/jobs/16124986305) | 17 | [2793285075](https://gitlab.com/gitlab-org/ops/artifact-registry/-/pipelines/2793285075) | step-11 MR, merge ref | 16:21 | | [16126491939](https://gitlab.com/gitlab-org/ops/artifact-registry/-/jobs/16126491939) | 16 | [2793530010](https://gitlab.com/gitlab-org/ops/artifact-registry/-/pipelines/2793530010) | step-11 MR, train ref | 17:19 | | [16127832236](https://gitlab.com/gitlab-org/ops/artifact-registry/-/jobs/16127832236) | 16 | [2793740112](https://gitlab.com/gitlab-org/ops/artifact-registry/-/pipelines/2793740112) | `main` at `936f54ea2` | 18:26 | | [16128015324](https://gitlab.com/gitlab-org/ops/artifact-registry/-/jobs/16128015324) | 18 | [2793767904](https://gitlab.com/gitlab-org/ops/artifact-registry/-/pipelines/2793767904) | step-4 MR, merge ref | 18:37 | The step-11 rows belong to gitlab-org/ops/artifact-registry!1965+s, whose train car the 17:19 failure dropped at 17:40 and which merged at 18:26 after two more pipelines. The step-4 rows belong to gitlab-org/ops/artifact-registry!1994+s, whose diff is six `internal/managementapi` files and whose branch does not contain the failing test file at all: a merge-ref pipeline runs the target's tests, so an open merge request inherits `main`'s flake even when its own packages are untouched. The 18:26 row is the one to read twice. That pipeline runs `main` at `936f54ea2`, the merge commit of the very change that introduced the test. ## The pin sits below every scale anyone has measured The fixture seeds two rows in the namespace under test and one row in a second namespace. gitlab-org/ops/artifact-registry#564+s measured this walk from 100 to 1,000,000 namespace blobs and 1 to 512 digests on PostgreSQL 16, 17, and 18, and found `blob_storage_blobs_by_namespace_pNN_pkey` serving it in every cell, with the narrow covering index never chosen. The fixture sits below the smallest namespace that grid covers, and the grid says nothing about two rows. The test's own doc comment concedes that the shape moves with version and cardinality: "PG18 reaches it per digest past 1M rows; PG16 and PG17 scan the whole partition and join above it, to about 100 rows before a bitmap heap scan." The assertion pins the primary-key shape anyway, at a row count where that comment expects a different one. #564 did find cardinality dependence inside its grid, and it is a different one: in 10 of its 41 cells the planner switches the join method to a merge or hash join over the one partition's namespace slice, while still reaching the primary key index-only. The shape these failures show is not a cell that grid covers. It is the shape a fixture-scale measurement on the step-11 branch had reported, which #564 refuted for 100 rows and up and left open below that. ## What #872 covered, and what it left gitlab-org/ops/artifact-registry#872+s closed on 2026-08-26 at 15:35 UTC as fixed in gitlab-org/ops/artifact-registry!1952+s, which reseeds the deep-page sibling into its own repository for `TestContainerRemoteTagStore_DeepPageIsIndexBacked`. That is a different test with a different fixture, so the class outlived the issue and this member of it has no owner. The fix direction #872 recorded does not transfer either. "ANALYZE the seeded tables before the EXPLAIN" is already what this test does, through `testutil.AnalyzeBeforeExplain(t, db, "blob_storage_attachments", "blob_storage_blobs_by_namespace")`, and that helper predates #872: it landed with S14 Step 6, `828d66708`. The test flips with the ANALYZE in place. ## Candidate fixes Three options, none chosen here: - **Widen the pin** to accept every shape the planner legitimately reaches at fixture scale, keeping the assertions on what the read must not do. - **Force the planner in the fixture**, either with session settings (`enable_bitmapscan`, `enable_seqscan`) or with a seed large enough to reach the regime the pin describes. Both trade fidelity for determinism, and the larger seed costs integration-suite wall clock. - **Re-scope what the pin proves**: keep the single-partition assertion and the no-heap-filter assertion, which are the prune guarantees the walk owes, and drop the index identity, which #564 already measures at production scale. ## Done when - `test:integration` passes the test on PostgreSQL 16, 17, and 18 over repeated runs, rather than on one roll. - The option taken is recorded here with its reasoning, so the next planner flip is read against a decision rather than rediscovered. Related to https://gitlab.com/gitlab-org/ops/artifact-registry/-/issues/316 --- > 🤖 **Triage note** — This issue has been classified as `type::maintenance` and labelled with `Category:Artifact Registry`, `devops::package`, and `group::package registry`. > > **Reasoning:** This is a flaky integration test caused by a PostgreSQL query planner plan-shape mismatch at low fixture cardinality. The work involves fixing or hardening a test assertion — no user-facing functionality is broken or being added. It is squarely a test/CI reliability improvement, which falls under `type::maintenance`. > > If this classification doesn't look right to you, please update the labels and let the team know!