## Original finding, 2026-08-24 Measured 2026-08-24: median 104.90s, min 101.08s, max 105.81s, 3 observations, all passing. Package `internal/datastore/migrations`. Very stable, unlike most rows. The slowest stable passing test in the project, and the only one whose cost grows structurally: it migrates the schema up, down, then up again, so every migration ever added makes it slower, forever. `TestWidenFormatEnumMigration_UpRejectsContainerRows` (35.39s) and `_DownRejectsOciRows` (29.93s) are the same shape, for about 170s combined. ## Analysis, 2026-08-27: the test is not the lever, the trigger is Since the original finding was written, !1561 (merged 2026-08-17) split this package into its own job, `test:integration:migrations` (`.gitlab-ci.yml:1025`). That changes what this item should do. ### How long the job takes, and a disagreement worth resolving first Full census of the job over 2026-08-14 to 2026-08-27, snapshot 2026-08-27: 3,905 pipelines, **11,723 job legs**, success n=11,311, **p50 674s**, p95 785s, max 1,205s. PG18 runs ~10% slower than 16/17 (p50 723s) and sets the family's finish time. Two measurements of `TestMigrations_UpDownUp` are on record and they do not agree: | Source | Figure | Measured | Method | |---|---|---|---| | This item's original finding | 104.90s (min 101.08, max 105.81, n=3) | 2026-08-24 | gotestsum per-test durations over 40 sampled job runs | | Epic gitlab-org&23241 | 387s, inside an 870s package total | 2026-08-13 | CI job-trace profile of pipeline 2758016461, verified against a local PostgreSQL | Against the measured job p50 of 674s that is either ~16% or ~57% — the difference between "bounding this test cannot fix the job" and "it mostly can". Neither figure is obviously wrong: the epic's profile predates !1561's durability-off GUCs and the job split, and the chain has grown from 75 migrations to 92 in the interval. But #175 records durability-off as worth only ~2.5% on the reset, so the GUCs alone do not explain a 3.7x gap. **Resolving this is one gotestsum run of the job at head, and it should happen before anyone sizes the in-job work.** It does not affect this item's own argument: a run that never happens costs zero under either figure. ### The job runs on 63.6% of merge requests; 4.4% of them touch migrations `test:integration:migrations` runs exactly one package but reuses `rules: *integration-rules` verbatim (`.gitlab-ci.yml:1057`), the anchor belonging to `test:integration` (`.gitlab-ci.yml:859`). The in-file comment describes this as "same trigger surface". The breadth was inherited by reuse; !1561's review discussed the Postgres GUCs, not the rules. So a change under `internal/format/npm/` fires all three PG legs of a schema-only suite. Over the 1,683 first-parent merges on `main` from 2026-03-04 to 2026-08-25: | Population | MRs | Share | |---|---:|---:| | Match the current broad `changes:` list — job runs today | 1,071 | 63.6% | | Touch `internal/datastore/migrations/**` at all | 74 | 4.4% | | Add or edit a file under `internal/datastore/migrations/sql/` | 60 | 3.6% | Among the 479 merge-request runs a migrations-scoped list would drop, the biggest contributors are `internal/datastore` outside `migrations/` (188 MRs), `cmd/artifact-registry` (152), and `internal/format/npm` (101). Runs by arm, same census: | Arm | Legs | Share | |---|---:|---:| | Merge request pipelines (broad `changes:` list) | 8,093 | 69.0% | | Branch push with no open merge request (same list) | 2,043 | 17.4% | | Push to `main` (unconditional) | 1,149 | 9.8% | | Tags (unconditional, 145 distinct tags) | 435 | 3.7% | Total runner time for the family: **1,970.6 h over 13.2 days, ~149 runner-hours per day.** ### Wall clock: small today, large the moment the sharding merge request lands Measured today the job is the pipeline ceiling in only **107 of 3,905 pipelines (2.7%)**, finishing a median 373s before the real ceiling, which is `test:integration: [POSTGRES, 18]` (2,088 pipelines, 53.5%). The !1561 split achieved what it claimed. !1781 shards `internal/datastore` inside `test:integration`. While that merge request is open, the ceiling stays `test:integration: [POSTGRES, 18]` and narrowing this job's trigger buys runner cost but little wall clock; once it merges, `test:integration:migrations: [POSTGRES, 18]` becomes the ceiling — #622 records this and the accompanying measurement of 12.58 min median. The value of the change below therefore rises on that merge, and neither change blocks the other. ## Overlap with epic gitlab-org&23241 @hswimelar owns [epic 23241](https://gitlab.com/groups/gitlab-org/-/work_items/23241), "Artifact Registry: cut PG integration test time by ~45% (migration squash before closed beta)". Its six children, #618 through #623, are all open and unassigned as of 2026-08-27. **The epic and this item are orthogonal levers on the same job: the epic reduces what one run costs, this item reduces how many runs happen.** They multiply rather than overlap, and neither blocks the other. At the epic's own expected end state of 11 to 12 minutes per leg, the job would still run on 63.6% of merge requests. Where they touch: | Epic item | Relation to this item | |---|---| | #620, squash the chain to a generated baseline | **Supersedes this item's original scope.** It resets the growth term and retires the version-anchored guard tests below the floor, which is both `TestWidenFormatEnumMigration_*` tests. No separate work on those belongs here. | | #622, re-measure and prune residual migration-suite items | **Closest overlap.** It already names "running the `UpDownUp` chain walk on one PG leg or in a parallel job", which is this item's original "move it off the critical path" option. That option belongs to #622; this item drops it. | | #618, #619, #621 | No overlap: `internal/datastore` fixtures and `cmd/artifact-registry` CDN boot tests, reached by other jobs. | | #623, re-squash cadence | No scope overlap, but it edits the same chain-run policy section of `docs/dev/go-testing.md` that part 3 below touches. Coordinate the edit. | Nothing in the epic changes **when** the job runs. That is the whole of what is left for this item, and it is what the retitle reflects. ## Suggested path Four parts. The first is the lever; the rest are what has to move with it. ### 1. Give the job its own `changes:` anchor, and drop the tag arm Stop reusing `*integration-rules`. The suite's real input surface is wider than `sql/`: it imports `internal/testutil` (6 files) and `datastore.NewMigrationClient`, and it links `github.com/pressly/goose/v3` v3.27.3, LabKit v2 and pgx, so the toolchain and module files have to stay. Proposed list: ```yaml - "internal/datastore/migrations/**/*" - "internal/testutil/**" - "internal/datastore/*.go" # datastore.NewMigrationClient, non-recursive on purpose - "go.mod" - "go.sum" - ".tool-versions" - ".gitlab-ci-asdf-versions.yml" - ".gitlab-ci.yml" ``` **Keep the unconditional `if-default-branch` arm.** It is the backstop that makes narrowing the merge-request arm safe, and at 383 `main` pipelines in 13 days (~29/day) it already runs the full up-down-up far more often than a nightly schedule would. This retires the original "run the full up-down-up nightly" option: `main` is strictly better, and no test schedule exists today (four schedules exist; all are sync or Renovate jobs). **Drop `if-tag`.** Every recent tag sits on `main`, so a tag run re-tests a commit `main` already tested. `.gitlab-ci.yml:1112` sets the precedent in this same file: "No if-tag: the identical code already ran this job on the MR and merge pipelines, so a release-tag rerun adds no coverage." Measured reduction on the merge-request arm, from the 1,683-merge history: | Variant | Reduction in the job's MR runs | |---|---:| | List above | 44.7% | | Same, minus `.gitlab-ci.yml` | 61.6% | | `migrations/**` alone (too tight — drops real inputs) | 93.1% | `.gitlab-ci.yml`, `go.mod`/`go.sum` and the two toolchain files account for **518 of the 592** hits the narrow list keeps, so the choice about `.gitlab-ci.yml` is most of the difference between the first two rows. Estimate, not a measurement: applying the merge-request-measured rate to the branch-push arm (a different population) puts the first variant at ~39% of all 11,723 legs, ~771 runner-hours per 13 days; the second at ~53%, ~1,063 h. Dropping tags adds 435 legs, ~74 h. ### 2. Stop excluding the package from `go_unittests` and `test:race` `GO_UNITTESTS_EXCLUDE_PACKAGES_REGEXP` (`.gitlab-ci.yml:21`) excludes `/datastore/migrations$` on the rationale that the package is integration-only and "shows as EMPTY/0%". That is now false: `runner_test.go` and `migrations_checksum_test.go` hold **23 untagged tests that pass in 1.2s with no database** (verified by `go test -list` on `c410cea46`). `test:integration:migrations` is currently their only home in CI, so narrowing its trigger silently narrows theirs. Removing the exclusion fixes that and costs ~1s. ### 3. Correct the CI table in the migrations doc `docs/dev/database-migrations.md:129-144` does not mention `test:integration:migrations`, and its `test:integration` row is already wrong: that job grep-excludes the migrations package at `.gitlab-ci.yml:840`. ### 4. Say in the merge request description that nothing in CI guards this `scripts/ci/check-integration-test-wiring.sh` offers no protection for this decision. It is file-global: it pools every `-tags=integration` package token and every `dir/**` glob in `.gitlab-ci.yml` into two flat sets and asks only whether each integration-tagged directory is named somewhere and glob-covered somewhere. It never associates a job's package list with that job's own `rules:`. It stays green under any narrowing here, including deleting the job. ## What coverage moves from the merge request to `main` Left uncovered on merge requests, on purpose: a change under `internal/datastore/**` but not `migrations/**` that invalidates a schema assertion. It is caught on `main` within one merge instead of on the merge request. The transitive compile closure (`internal/remote`, `internal/storage`, `internal/jobsriver`, and the rest, all reached through `internal/datastore`) keeps its cover from `go_unittests`, `test:race` and `test:integration`, which retain the wide anchor. Unaffected either way, because they read `*changes-migrations` or `*changes-sql-migrations` already: `db:migrate`, `db:structure-check`, `jet:generate-check`, `lint:migration-ordering`, `lint:migration-immutability`, `lint:sql-format`, `lint:migrations`. ## Sequencing - Any change here edits `.gitlab-ci.yml`, which !1781 also rewrites. Whichever lands second conflicts, and no pipeline reports it. Coordinate with #751, which owns !1781. - Epic gitlab-org&23241 needs no sequencing against this item beyond the two shared edits named in the overlap table above. Parent: https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/953