Flaky test:integration:migrations: any PostgreSQL leg can exhaust its 23-minute timeout and report tests it never ran
What happened
test:integration:migrations: [POSTGRES, 16] failed on merge request !2268 (merged) with a wall-clock timeout, not an assertion.
panic: test timed out after 23m0s
running tests:
TestWidenFormatEnumMigration_UpRejectsContainerRows (6s)The job passes -timeout 1380s. The package exhausted it, the test binary died, and gotestsum then reported 76 of 1379 tests as (unknown). None of the 76 is a real failure: every one of them was still parked in t.Parallel() when the alarm fired. A reader who opens the job sees 76 red lines and no failing assertion, which is the part that costs the most time to diagnose.
- Failed leg: job
16266545752, 1508s, pipeline2814382885. - Retried with no code change: job
16267088666, 833.9s, passed. Pipeline went green.
It is runner-speed variance, not the branch
The branch under test adds and edits no file under internal/datastore/migrations/.
Three legs of the same suite, same commit, same runner class k8s.saas-linux-medium-amd64, same pipeline:
| Leg | Duration | Result |
|---|---|---|
[POSTGRES, 17] |
406s | success |
[POSTGRES, 18] |
963s | success |
[POSTGRES, 16] |
1508s | failed on the 1380s timeout |
[POSTGRES, 16], retried |
834s | success |
On main at 89fa85ea8, which is this branch's own merge base, the PG16 leg took 634s and passed.
The headroom is thinner than the census suggests
Issue #955 (closed) carries a full census of this job over 2026-08-14 to 2026-08-27: 11,723 legs, p50 674s, p95 785s, max 1205s.
Against the -timeout 1380s, that maximum leaves about 14% headroom. The 1508s leg above is past the recorded maximum, so the distribution's tail reaches the ceiling. A leg that crosses it does not fail slowly or partially: it takes the whole pipeline red and reports 76 tests it never ran.
The cost also grows on its own. The suite migrates the schema up, down, and up again, so every migration ever added makes every future leg slower. #620 records that growth term and proposes squashing the chain to a generated baseline.
What this issue is, and is not
It is not a duplicate of these, and each is worth reading first:
- #955 (closed), closed. Owns when the job runs — narrowing its
changes:trigger. Its census is quoted above. It does not address the timeout ceiling. - #747 (closed), a different flake in the same job: the
container_imagespartial-index plan subtest flips its assertion. That one produces a real failing assertion; this one produces a dead binary. - #622, open. Re-measure the suite after the migration squash. The observation here is a data point for it, but the ceiling is reachable today, before any squash.
- #620, open. Squash the chain to a generated baseline, which is the structural fix for the growth term.
- #953, open. Pipeline duration generally.
Possible directions
Stated as options with their costs, not as a recommendation.
- Raise
-timeouton this job alone. Cheapest, and it converts a red pipeline into a slow one. It buys nothing against the growth term, and it widens the window in which a genuinely hung test burns a runner. - Lower the ceiling instead: land #620. Removes the growth term rather than accommodating it, and retires the version-anchored guard tests that dominate the tail. Much larger, and it is already owned.
- Split the two
TestWidenFormatEnumMigration_*tests off the critical path. #955 (closed) measured them at about 170s combined. It helps only if they are the tail, which the two measurements on record in #955 (closed) disagree about — that disagreement is unresolved and is onegotestsumrun away from being settled. - Do nothing and retry when it fires. Which is what happened here, at the cost of one diagnosis and one 23-minute leg.
How to reproduce
Nothing to reproduce deterministically: the trigger is runner speed. The signature to match when it recurs is panic: test timed out after 23m0s in test:integration:migrations, with a large (unknown) count and no failing assertion.
This is a bot message