ci: shard the internal/datastore integration suite (!1781) · Merge requests · GitLab.org / ops / Artifact Registry

What

Splits the top-level internal/datastore package out of test:integration and shards it four ways across the existing three PostgreSQL versions, plus a single job that merges the shards' coverage into one total.

Job	Instances	Change
`test:integration`	3	Loses `internal/datastore` (keeps `internal/datastore/schemas`)
`test:integration:datastore`	12	New: 4 shards x 3 PG versions
`test:integration:datastore:coverage`	1	New: merges the 12 profiles, reports the one coverage number

Why

test:integration set the critical path on any pipeline that ran integration tests: 17.3 minute median on the PG 18 leg across the 8 successful merge_request_event pipelines sampled on 2026-08-20. Parsing the trace of pipeline 2776028427 per package:


Whole run	`DONE 17637 tests, 31 skipped in 858.853s`
`internal/datastore`	spans 30s to 851s
Every other package	finished by 313s
Sum of the datastore tests' own durations	821s, against an 817s wall span

1.01x parallelism. The package ran strictly serially for about 9 minutes while three of the runner's four cores sat idle.

That is deliberate: internal/datastore/postgres_integration_test.go:3 carries //nolint:paralleltest // integration tests share a database. Making the suite parallel inside one binary is a harness refactor. Sharding across jobs sidesteps it, because each shard gets its own PostgreSQL service and TestMain already calls testutil.SetupIsolatedTestDB. No test code changes, no shared-database risk.

There is precedent: internal/datastore/migrations was carved out of this same job for this same reason.

Three ways this goes wrong silently, and what closes each

Every one of these ships green and saves nothing, or worse, loses coverage nobody notices.

1. The residual job would have run the suite twice

test:integration's filter was grep -vE '/datastore/(jet|migrations)(/|$)', which does not match .../internal/datastore itself. Left alone, the package runs in both the shards and the old job, the 9 minutes stay on the critical path, and the shards' speed makes it look fixed.

The corrected filter is grep -vE '/internal/datastore$|/datastore/(jet|migrations)(/|$)'. Both the /internal/ prefix and the $ anchor are load-bearing: /datastore$ alone would also strike any future package path ending in /datastore, and dropping the anchor takes the subpackages with it.

internal/datastore/schemas is the package at risk. After the correction it is the only datastore package test:integration still runs, so an anchor mistake drops it silently. scripts/ci/check-integration-test-wiring.sh cannot catch this: it harvests the ./package arguments off -tags=integration lines and never sees the grep, so it stays green either way. Verified by hand, the corrected filter leaves exactly one package standing:

gitlab.com/gitlab-org/ops/artifact-registry/internal/datastore/schemas

2. Eleven of twelve coverage profiles would have vanished

A job that needs: a parallel: job depends on all its instances and downloads all their artifacts into one workspace. Per the GitLab docs: "If the artifacts have the same name, they overwrite each other and only the last one downloaded is saved."

So the profiles are named coverage/datastore-pg${PG_VERSION}-shard${SHARD}.out and the aggregator asserts it received exactly 12 of them. Without that assertion a missing artifact yields a lower coverage number on a green pipeline, which is the failure mode nobody reviews.

3. The pipeline coverage number would have fallen for no reason

This is the design decision the issue asked to be made explicitly rather than by omission.

GitLab's pipeline coverage is the unweighted arithmetic mean of every job's coverage: value, and a parallel:matrix instance is its own row in that mean. Confirmed on pipeline 2776028427: 15 jobs report coverage, their mean is 74.7533, and the pipeline reports 74.75. test:integration contributes its value three times, once per PG leg. Twelve shards each reporting a quarter of the package would enter that mean twelve times.

Demonstrated locally on a smaller package, splitting one suite into two shards:

	Reported
Shard 1 alone	9.8%
Shard 2 alone	10.0%
Mean of the two, which is what the pipeline would show	9.9%
Merged profile	11.4%
Ground truth: both halves in a single run	11.4%

So the shards carry no coverage: regex and no artifacts:reports:, and test:integration:datastore:coverage reports the one merged total. Cobertura and the coverage: regex are independent (a job can upload a report without feeding the mean), so this loses no MR diff coverage.

Merging is concatenation with the duplicate mode: headers stripped, and that is correct rather than merely convenient. Both consumers fold repeated blocks: x/tools' cover.ParseProfiles behind go tool cover ORs the counts in set mode, and gocover-cobertura's AddOrUpdateLine does the same per line. The shards pass no -covermode and no -race, so the mode is set everywhere.

Plain cat is not a merge, and it fails asymmetrically, which is why the script does not use it. go tool cover on a naively concatenated pair reports:

cover: line "mode: set" doesn't match expected format: couldn't parse "Count": strconv.Atoi: parsing "set": invalid syntax

It rejects the stray second header outright; gocover-cobertura skips it and emits a plausible report. Same mistake, loud in one step and quiet in the next.

No new tool is pinned. gocovmerge, the usual answer, has had no commit since 2016-03-31 and no tags, and it is a thin wrapper over the same ParseProfiles.

Why the aggregator costs no wall clock

It sits in stage: validate alongside the shards, not a later stage, so it waits only on the shards rather than on all 30 to 70 validate jobs. A same-stage DAG edge is already proven in this pipeline: conformance:maven:s3-garage needs: build:conformance-ar, and on pipeline 2776028427 it was dispatched 3.4s after its dependency and the two-hop chain totalled 5.1 min against a 16.6 min pipeline.

Budgeted at: slowest shard roughly 2 min setup plus 821s/4 plus tail, so 6.5 to 7.5 min, and the aggregator 2 to 3 min. Measured at 8.0 min and 58s, so about 8.9 min against a ceiling of 13.5 min. Speeding up test:integration:migrations would make this chain the new ceiling, which is what #622 covers.

Corrections to the issue's own analysis

Three of the issue's stated facts did not survive verification. Recording them here rather than silently substituting the corrected version.

"Every locally defined job carries needs: [], so the validate stage is a flat fan-out." 45 of 46 do. .gitlab-ci.yml:2546 is the exception (.conformance-maven needs build:conformance-ar), which is what makes the aggregator's shape a proven pattern here rather than a new one.
"CI_NODE_INDEX and CI_NODE_TOTAL are set only under parallel: N, not under parallel: matrix." They are set under both; Gitlab::Ci::Config::Normalizer::MatrixStrategy emits the same instance and total attributes. The issue's conclusion holds for a better reason: under a 3x4 matrix CI_NODE_TOTAL is 12 and CI_NODE_INDEX enumerates the whole cross-product rather than the shard dimension, and the index shifts whenever the matrix definition is edited. So the shard index still has to be an explicit matrix variable.
"1293 names, three short of the trace's 1296, reconcile before relying on the split." Reconciled, and nothing is silently dropped. The invariant is "grep hits = runnable tests + 1", the +1 being TestMain (harmless in the alternation: the generated test main calls it before any -run filtering). 1293 hits means 1292 runnable tests, measured at 28b80f1b, the tree the issue was written against; 1296 runnable tests corresponds to 1297 hits, which main had at five commits that landed the same day. The gap is 4 tests of revision drift minus the TestMain over-count. main currently yields 1327.

Separately, gotestsum's DONE N tests counts subtests, so it can never be compared against a top-level name count at all.

What the shard script guards, and why each guard is there

scripts/ci/datastore-test-shard.sh, rather than an inline script: block, so shellcheck and shfmt cover it.

Empty-match is a hard failure. The issue's snippet assigned the whole pipeline's output to a variable, and the exit status of that pipeline comes from sort (0), not from git grep (1 on no match). An empty list would produce a -run pattern that matches nothing, run zero tests, and exit 0. The repo already documents this exit-code trap in scripts/ci/check-jet-null-wrap.sh.
The pathspec is :(top,glob). The plain :(glob) form resolves against the working directory, not the repo root. The same script documents the same bite.
A fuzz target aborts the run. A -run whitelist enumerates Test functions, so a FuzzXxx added to this package would stop running here without failing anything. docs/dev/go-testing.md actively recommends fuzzing for parsers, so this is a live forward hazard. There are none today.
^(...)$ anchoring. -run matches substrings by default and the package has hundreds of names that are prefixes of other names (TestBlobStorageAttachmentStore_Delete against ..._DeleteIfUnreferenced), so unanchored shards would overlap heavily. Every captured name is [A-Za-z0-9_]+ by construction, so no name can carry a metacharacter needing escape.
SHARD_TOTAL drift is caught from both sides. Too small, and the script rejects a SHARD outside 1 to SHARD_TOTAL. Too large, and scripts/ci/datastore-partition-check.sh fails, because the union of the names the shards selected no longer equals the package's full list.

An earlier revision of this bullet credited that second direction to the aggregator's profile count. That was wrong. The profile count is the PostgreSQL version count times the length of the matrix's SHARD list and never reads SHARD_TOTAL, so the two are independent. Measured on this branch's tree: with SHARD_TOTAL at 6 and the SHARD list left at 1 to 4, every shard passes its range check, all 12 profiles arrive, EXPECTED_PROFILES matches, and 598 of 1793 test names run in no job at all on a green pipeline. The partition check is what actually closes it, and it names the missing tests rather than reporting a count.

Partition re-derived after merging main: 448 + 449 + 448 + 448 = 1793, and concatenating the four shards reproduces the sorted input exactly. The count moved from 1327 because main gained datastore tests while this branch was open; every figure in this section is the post-merge one. The largest -run pattern is 24,540 bytes. The binding limit for one argv entry on Linux is MAX_ARG_STRLEN, 131,072 bytes, not the 1,048,576 ARG_MAX total an earlier revision quoted, so the headroom is about 5x and is reached at roughly 9,500 names. Still ample, and it shrinks as the package grows, so it is the number to re-check rather than to quote once.

Cost

12 jobs replace the datastore share of 3, each provisioning and migrating its own database.

Method for every figure below. Two green pipelines carry this branch's shape (2796387105, 2796795149). The baseline is the six most recent successful merge_request_event pipelines with a comparable job count, 73 to 86 jobs, so the job set is close to this MR's own: 2796652584, 2796606194, 2796578487, 2796543631, 2796480744, 2796466120. A wider sweep of 29 successful MR pipelines gives the same test:integration median to within 0.7 min.

gitlab-advanced-sast is the control. This MR does not touch it, so when it moves between two runs the fleet moved, not the code. It reads 12.0 min on the first of the two shard pipelines and 18.8 on the second, which is why those two disagree on everything else by roughly the same factor. Quote the range, not whichever end flatters the change.

	job-minutes
12 shards	+58.2 to +69.5
aggregator	+1.0 to +1.1
saved on the three `test:integration` legs (66.1 median, down to 21.2 to 29.1)	-37.0 to -44.9
net per integration pipeline	+14 to +34

Whole-pipeline job-minutes barely move: 261.3 median before against 243.3 and 290.5 after. That comparison is weak, because an MR pipeline's job count varies with which files changed, so read the datastore block above instead.

Issue #751 budgeted "on the order of 30 or more". The measured figure lands inside that, at the top of the range in the contended run and well under it in the quiet one. An earlier revision of this section reported +49.7 from a single pair of pipelines on a tree with 1327 tests. That number was too high, and it is worth saying why the correction goes this direction: the package has since grown to 1793 tests, so the cost should have risen, not fallen. One pipeline per side was the problem, not the arithmetic.

Two further cost notes:

The go-build cache is keyed on CI_JOB_NAME_SLUG, so each of the 12 matrix instances gets its own slot and all 12 are cold on the first pipeline after this merges. They compile identical binaries. Sharing one warm slot would need a single designated writer, which is the pattern the go-mod cache comment describes; not done here, and worth revisiting if the cold cost shows up.
43% of the sharded work is already duplicated elsewhere. Of the 1793 names, 771 sit in the package's 111 untagged test files, which go_unittests and test:race already run (internal/datastore is not in GO_UNITTESTS_EXCLUDE_PACKAGES_REGEXP). The other 1022 are in the 153 integration-tagged files and are unique to this job, 1021 of them runnable once TestMain is discounted. Re-derived on this branch and cross-checked against scripts/ci/datastore-test-names.sh; an earlier revision had 555 and 771 against a 1326 total, from before main grew the package. They are sharded anyway, matching the issue, because they compile into this binary and filtering them out of the -run regex would stop them running here, where the tagged TestMain has provisioned a database that the unit binary has not. Narrowing the shards to the tagged half is a real further win and a separate change, since it needs an argument that no untagged test depends on the integration binary's setup.

The win is capped, and the cap needs a home

Once test:integration drops, the ceiling for integration pipelines becomes gitlab-advanced-sast, and it is not a close call. On both shard pipelines it is the single longest job and it equals the whole pipeline's wall clock: 12.0 min of a 12.0 min pipeline, then 18.8 of 18.8. test:integration:migrations sits 6 min below it at 12.7. !1779 (merged) in this series addresses the SAST half.

An earlier revision called SAST "marginally above" migrations, from a 12.64 against 12.58 median. Those two numbers are within noise of each other and both were taken before this branch removed the job that used to hide them. Measured on this shape, SAST is the ceiling on its own and migrations is not near it.

The issue asks whoever opens this MR to file an issue for the migrations half so the cap has a home. It already has one: #622, "Re-measure the PG integration suite after the migration squash and prune residual chain costs", which names the migrations package's TestMain head-template and the UpDownUp chain walk as the two residual costs. Rather than file a duplicate I have commented there with what this MR changes about its payoff: until this lands, shaving the migrations suite buys nothing because test:integration is 4.7 min longer and hides it; afterwards it is the pipeline's wall clock.

Note also that test:integration: [POSTGRES, 17] already sits at a 12.05 min median, so that leg's win is close to zero. The gain is concentrated in the PG 16 and PG 18 legs.

Before and after, measured

	Before (median, n=6)	After (n=2)
`test:integration: [POSTGRES, 18]`	23.2 min	7.4 and 12.1 min
`test:integration: [POSTGRES, 17]`	22.1 min	6.6 and 9.6 min
`test:integration: [POSTGRES, 16]`	20.6 min	7.2 and 7.5 min
Slowest of the 12 shards	n/a	7.8 and 7.3 min
Datastore critical path (slowest shard + aggregator)	inside a 23.2 min job	8.8 and 8.4 min
`test:integration:datastore:coverage`	n/a	1.0 and 1.1 min
Whole pipeline	23.6 min	12.0 and 18.8 min
Pipeline coverage	75.38 to 75.41%	75.75 and 76.44%
Longest job	`test:integration` in 5 of 6	`gitlab-advanced-sast` in 2 of 2

Five things to read out of that.

The cleanest single number is the test:integration leg, not the wall clock. It is the same job with the same rules: on both sides, so the comparison is direct, and it is stable: the 29-pipeline sweep puts the slowest leg between 21.5 and 25.0 min. Wall clock is not stable. The same sweep's pipelines range from 22.4 to 46.0 min while their test:integration leg barely moves, so a wall-clock pair proves much less than it looks like it does.

The aggregator is cheaper than budgeted. I estimated 2 to 3 min and it takes about a minute, so the shard-plus-aggregate chain is 8.4 to 8.8 min. It no longer sits on the critical path at all, because test:integration still takes longer than that even after losing the package.

The coverage number went up, not down. That is the point of the aggregator design. test:integration:datastore:coverage reports 89.3% for the merged profiles, which is above the pipeline mean, so adding it as one row lifts the average from 74.75 to 75.76. Twelve partial shard totals would have done the opposite. test:integration reports 91.0% on each leg, up from 90.4 to 90.5% across the four sampled baseline pipelines: the package it lost sat at 89.2%, below the residual mix, so removing it lifted the remainder slightly. An earlier revision said "unchanged", which is the sentence a reviewer would have used to conclude nothing moved.

The residual job is confirmed correct. Its trace shows internal/datastore only inside the go list arguments and the filter, and internal/datastore/schemas is the one datastore package it still compiles.

The new ceiling is migrations, as predicted, and it is the whole remaining story. Longest jobs on this pipeline:

Job	Duration
`test:integration:migrations: [POSTGRES, 18]`	812s
`gitlab-advanced-sast`	757s
`test:integration:migrations: [POSTGRES, 16]`	686s
`test:integration:migrations: [POSTGRES, 17]`	646s

test:integration no longer appears in the top six.

One thing the numbers expose that the design did not anticipate: the shards are unbalanced. Across the 12 instances the spread is 2.36x on one pipeline (3.3 to 7.8 min) and 1.63x on the other (4.5 to 7.3 min), because round-robin over sorted test names balances name count rather than duration.

Which shard is slowest is not stable, so do not tune for one: shard 3 is slowest on 2796387105 and shard 4 on 2796795149. An earlier revision said shard 3 is "consistently" slowest, on one pipeline's evidence. The slowest shard sets the chain, so a duration-aware split would buy another 2 to 3 minutes. Not done here: it needs a persisted timing profile, which is a different mechanism with its own staleness problem, and the current split already fits under the ceiling.

Stack

!1930 targets this branch rather than main, and !1931 and !1932 stack above it. In merge order:

MR	Target	What it does
!1781 (this one)	`main`	Shards `internal/datastore` out of `test:integration`
!1930	this branch	Adds `--junitfile` to every project-owned gotestsum invocation
!1931	!1930	Coverage badges
!1932	!1931	Local HTML coverage task

This branch is merged with main rather than rebased onto it, so !1930's diff stays valid. A rebase here would invalidate it.

Conflicts

.gitlab-ci.yml is touched by four other open MRs, checked per the CLAUDE.md guardrail 23 search on 2026-08-20 and re-checked on 2026-08-27. The operator was consulted and asked for this MR to be opened regardless.

!924 (draft, @jay_mccure, last updated 2026-07-27, carries merge conflicts) replaces the hand-rolled coverage plumbing with the go-test framework's. That is directly on this MR's subject, so whichever lands second has real work to do, not just a textual rebase.
!818 (closed) (draft, @jay_mccure, last updated 2026-07-06) renames test:integration to test:go:integration. That rename would have to reach the new jobs and the two scripts.
!1638 (merged) (ready, @Shweta, updated 2026-08-20) adds a Caproni e2e job. A different region of the file; a textual rebase at most.
!1782 (merged) (ready, same author, opened five minutes after this one) sets default: interruptible: true. Missed by the original search because it did not exist yet. No textual conflict, and the semantics are checked: the 12 shards and the aggregator all inherit interruptible: true, which is the correct setting. Marking the aggregator false would be actively wrong, since it would survive while all twelve of its needs: were cancelled.

glab mr list --search 'shard' returns nothing, so nobody else has started this.

The conflict with main is resolved. main added ./internal/lifecycle/... to the same packages= line this MR rewrites, while this branch was open. Both edits are kept: the go list line names ./internal/lifecycle/... and carries this MR's /internal/datastore$ filter. main's test:lifecycle-failpoints job is also why check-integration-test-wiring.sh now refuses a literal -run, below.

Follow-up commit: the guards the first revision claimed but did not have

Four fixes, each with the failure it closes:

The partition check above. SHARD_TOTAL drift silently dropping a third of the suite was the headline defect, and the profile count never caught it. datastore-test-names.sh is now the single derivation of the package's test names, shared by the shard script and the aggregator so the selector and the check cannot drift apart. The same assertion also closes Example functions carrying an // Output: comment, which go test runs and -run filters exactly like a fuzz seed corpus and which the Fuzz guard did not mention.
go test -timeout was 1380s against a 20m job cap, so the runner killed a hung shard before Go could dump goroutine stacks. test:integration:migrations sizes its cap the other way round deliberately. Lowered to 900s.
The four scripts were absent from &integration-test-files, so an MR editing only a script changed which tests run and revalidated none of them.
check-integration-test-wiring.sh was vacuous for this package. It read only .gitlab-ci.yml and credited internal/datastore to test:integration's go list line, which that same line's grep -vE strikes out. It now folds line continuations, reads any scripts/ci script carrying the tag, and refuses to credit a package to a set that filters it out. Verified both ways: green today, red when the shard script stops naming the package.

Also check-pg-version-sync.sh, which compared the deduplicated union of every PG_VERSION list against the three versions, so one job dropping a version passed as long as another still listed it. Each list is checked on its own now, and the old form was confirmed to miss exactly that case.

The empty-shard guard was unreachable as written: printf '%s\n' "" emits one line, so wc -l returned 1 and the count could never be 0. Tests -z instead.

Testing

CI plumbing, so the test is the pipeline itself. What to check on this MR:

test:integration:datastore produces 12 green jobs, and each logs its shard N/4: X of Y test names line, with the four X values summing to Y and the same Y on every shard. Y is derived per commit, so read it off the job rather than against a number quoted here (1793 at the time of writing).
test:integration:datastore:coverage finds exactly 12 profiles and reports a total in the same range test:integration reported before (roughly 90%).
test:integration no longer runs internal/datastore but still runs internal/datastore/schemas.
The pipeline coverage number does not drop.

scripts/db/check-pg-version-sync.sh and scripts/ci/check-integration-test-wiring.sh both pass locally, and all four new scripts are shellcheck- and shfmt-clean.

Diff size

723 insertions, 43 deletions over 9 files, so 766 reviewable LOC. Past the 500 ceiling docs/dev/development-model.md sets, so per guardrail 22, the split and the reason it is not smaller:

Group	Files	+/-
CI config	`.gitlab-ci.yml`	+182/-6
New scripts	`datastore-test-shard.sh`, `datastore-test-names.sh`, `datastore-partition-check.sh`, `datastore-coverage-merge.sh`	+317
Meta-check rewrites	`check-integration-test-wiring.sh`, `check-pg-version-sync.sh`	+155/-26
Docs	`docs/dev/go-testing.md`	+64/-10
Hook config	`.pre-commit-config.yaml`	+5/-1

Two reasons a split would be worse rather than smaller.

The four new scripts are one mechanism. Landing the shard script without datastore-partition-check.sh ships the SHARD_TOTAL drift defect for a merge window, and that defect is the one that runs a third of the suite nowhere on a green pipeline. datastore-test-names.sh exists precisely so the selector and the check cannot be two derivations, so it cannot go in a different MR from either.

The meta-check rewrites are not drive-by. Moving internal/datastore's package set out of .gitlab-ci.yml and into a script is what makes check-integration-test-wiring.sh wrong, and adding a second PG_VERSION list to the file is what makes check-pg-version-sync.sh wrong. Both are caused by this change and would be unexplained on their own.

The diff is comment-heavy by design: the four new scripts are roughly half comment by line, and a large share of the .gitlab-ci.yml delta is the reasoning on the two new jobs.

Review fixes

Four commits on top of the two above, from a full review of the branch. Each names the failure it closes.

The wiring check went vacuous the moment this branch merged

main gained test:lifecycle-failpoints while this branch was open. Its go test line carries -tags=integration and names ./internal/datastore and ./internal/lifecycle, running two of their tests under a literal -run. check-integration-test-wiring.sh credited a package to any line that named it, so on the merged tree, deleting ./internal/datastore from datastore-test-shard.sh left the check green, and so did dropping ./internal/lifecycle/... from the test:integration go list line. Both were measured, both directions, before and after.

A literal -run now credits nothing: it enumerates a fixed set of test names, so it runs the package without covering it and cannot grow when the package does. A -run built from a shell parameter is still credited, because it is derived from the package rather than frozen against it, and proving it exhaustive belongs to whatever derives it. The script says that rather than implying it can tell an exhaustive derivation from a partial one.

Two smaller holes in the same check: it was a member of its own corpus, so its own failure message was one concrete example path away from vouching for a package forever (comment lines are now dropped and the discovery loop skips this file); and lint:integration-test-wiring triggered on .gitlab-ci.yml but not on the scripts holding the package sets it reads, so an MR editing only datastore-test-shard.sh ran neither the job nor the hook.

`check-pg-version-sync.sh` was rewritten into a fail-open guard

Two ways to say nothing, both measured against the previous version:

Case	Old	New (before this fix)	Now
No `PG_VERSION:` line at all	exit 1	exit 0	exit 1, with a message
Comment line reading `PG_VERSION:`	exit 0	exit 1, no output	exit 0

The first is the one that matters: the guard passed once the matrix it guards was gone. The second dies under set -e mid-loop and prints nothing, and .pg-version-matrix's own comment already reads # PG_VERSION drives the postgres service image tag, one colon short of it. Comment lines are excluded now, an empty match on either side is an explicit failure, and a list carrying a variable reference instead of literals is reported as such.

A build constraint other than `integration` would run nothing, quietly

datastore-test-names.sh derives names from the source, not the compiled binary, so a test file tagged anything else contributes its names to the -run alternation and to the partition check's expectations while go test -tags=integration compiles none of them. Both sides shrink together, which is the one drift a single derivation cannot see. Now a hard failure naming the constraint; verified by tagging one file integration && !race.

Job hardening

retry on runner_system_failure and stuck_or_timeout_failure. The leg went from 3 jobs to 13 and the aggregator needs all twelve, so one dying runner takes the coverage report with it.
artifacts: when: always, so the shard whose tests failed still uploads the manifest saying which tests it was holding.

Claims that were not true of the tree

Every one of these is a comment or a doc line that said something the code does not do. Listed rather than summarised, because each was independently wrong:

The new job's header credited check-pg-version-sync.sh with pooling and deduplicating PG_VERSION literals across the whole file. The second commit on this branch removed that sort -nu, and its replacement's own header says the opposite.
EXPECTED_PROFILES was called "PG versions x SHARD_TOTAL" in four places, contradicted six lines away in two of the same files. This MR already corrected that conflation once, in prose, and left the comments standing.
Two comments named datastore-coverage-merge.sh as the script that re-derives the name list. The assertion moved to datastore-partition-check.sh.
datastore-partition-check.sh claimed it "closes the same hole for a pathspec or a name filter that stops matching what it used to". It cannot: both sides call the same derivation. It now states what it does establish, and that it compares selections rather than results.
Its success line said "all N test names ran". It reads no test result.
The shard script said CI_NODE_INDEX/CI_NODE_TOTAL are set "only under parallel: N". They are set under parallel: matrix too. The conclusion holds for the reason this MR's own "Corrections" section already gives.
"hundreds of names that are prefixes of other names" measured 200 of 1793.
The go-build cache comment justified per-job slots by build tags, -race and GOFIPS140, none of which distinguishes the twelve matrix instances.
go-testing.md's coverage row read "Every integration-enrolled package except the two split out below", which omits nine packages covered by the Redis topology, driver and BLV jobs.
"The test:integration job covers everything regardless" was left in the same file, and stopped being true for internal/datastore in this MR.

Left as-is, deliberately

Twelve identical go-build cache slots. CI_JOB_NAME_SLUG embeds the matrix values and nothing about SHARD or PG_VERSION changes what compiles. Collapsing them needs a single designated writer, the go-mod pattern. Documented on the cache anchor now instead of only here.
Duration-unaware sharding. Shard 3 runs 1.6x shard 1. A duration-aware split needs a persisted timing profile with its own staleness problem.
The two commits are not squashed. The MR has squash-on-merge set, so main records one commit and the first commit's superseded claims do not survive into the history.

Related to #751

Edited Aug 27, 2026 by Dzmitry (Dima) Meshcharakou

ci: shard the internal/datastore integration suite