test(conformance): the S08 remote run's adversarial pass, and its closing housekeeping

Summary

The S08 remote goal run's adversarial pass, which the goal requires as a distinct activity after the last step: re-review every implemented test case hunting assertions that cannot fail, tautologies, catalog rows whose assertions do not cover their stated behaviour, and evidence gaps. Plus the run's closing housekeeping.

Related to #47 (closed).

Stacked on !273 (merged) (Step 37), which is stacked on !272 (merged), !271 (merged) and !266 (merged). d20c40e is pkg/conformance test-only and can be cherry-picked forward if the stack stalls. The citation rewrite genuinely must stack: it touches files the three OCI batches introduce.

The two findings

Both are the same shape, and it is the shape this run kept meeting: an assertion whose expected value is the thing under test. Review does not catch it, because "is this behaviour asserted?" answers yes. The question that finds it is "could this assertion's expectation move with the code it checks?", and the mechanical form of that question is a mutation.

3e9d77d — three OCI error codes were unfalsifiable

DIGEST_INVALID, NAME_UNKNOWN and BLOB_UPLOAD_INVALID. Changing any left the whole package green.

The other seven were safe by accident: a code is pinned when some test happens to serve it as a literal while production compares the constant. A row whose fake is built from the constant gets no pin, which errors_digest_invalid_test.go is exactly:

body := ociErrorEnvelope(errCodeDigestInvalid)
assertOCIErrorResponse(t, body, errCodeDigestInvalid)

The consequence runs the direction that is easy to miss. These are wire values, so a typo does not make the suite lenient — it makes the suite wrong about a correct registry, demanding DIGEST_INVALD from a conformant target and reporting a failure that target cannot fix.

Two tests, because one does not reach what the other covers. The set test exists because five of ociDefinedErrorCodeSet's fourteen entries are bare literals with no constant, which the sweep structurally cannot see — it mutates constants, and a literal inside a composite is not one. That set is what S07 AC #42 (closed)'s membership check runs against.

d20c40e — six of thirteen ConfigError.Field constants were unfalsifiable

registry-url, run-id, settle-timeout, priority, allow-redirect-host and repository-kind. Changing fieldRepositoryKind from "repository-kind" to anything left every test in the repository green. validate_test.go's table expects the constant (wantField: fieldRegistryURL), so production and test move together. The literals appear in that file only inside comments, and a comment is not an assertion.

The count is repository-wide and re-measured per constant, mutating each value and running ./pkg/... ./internal/... (3898aca7). The other seven were already pinned by a test asserting a literal against the live cfgErr.Field, and the file lists which test catches each, so the number is checkable without repeating the sweep.

This matters more than the error codes: ConfigError.Field is a documented public contract. S04 §Error types enumerates its values and S08's own amendment added four, so a library caller is entitled to match on the string. A drift is silent both ways — the caller takes its else branch, and the spec quietly stops describing the code.

The second test, distinctness, exists because the value table cannot catch a copy-paste giving two fields one value: that needs both rows wrong to be caught by comparison and none to be caught by distinctness. Not hypothetical for a family whose members are fieldUpstreamURL and fieldUpstreamFreeOnly and where four arrived in one change.

Both findings are pre-existing on main, not introduced by this run.

What the pass checked and did not find, which is also a result

• Catalog claims against test data. Every assertable token a row's Notes names must appear in that row's tests. The population is the non-optional rows of the two remote tables: 13 of the 15 oci.remote.* rows in docs/catalog/oci.md and 10 of the 12 npm.remote.* rows in docs/catalog/npm.md (cache-ttl and cache-invalidation are optional in both). Zero gaps.
• Evidence coverage. Every one of those same 23 rows has a reference control matrix. The single gap was already recorded: Step 19's m4-content-mismatch.
• Constant falsifiability, across oci, npm, maven and pkg/conformance. 108 survivors outside the ConfigError.Field constants, and none worth fixing — fixture-internal values, probe digests whose only property is being wrong, and message fragments. Unfalsifiable is the correct state for those.

That last ratio is the useful output. The technique is cheap and mostly returns noise, so its value is entirely in the triage, and a report claiming every unfalsifiable constant as a defect count would be worse than useless.

The housekeeping commit

3b63da5 stops committed code citing validation/, a directory the goal's own rule keeps out of every MR. The finding ID stays — it is the handle a reader can ask about, and oci.remote.blob-range not requiring Accept-Ranges is defensible only because R5 measured it absent — and only its presentation as a file in this repository goes. Comment-only: 26 files, 47 insertions and 54 deletions.

Its message justified doing this in one pass rather than per batch, because fixing the subset one MR happens to own would have left the same citation dangling elsewhere. That reason did not hold at the tree it landed on: 31 more Go files and both remote catalogs still carried one. ecfb86ac finishes the sweep under the same three rules (33 files, comment and catalog prose only), so the claim now holds at HEAD.

The review round

• dc3b3d88, cb0f4881: both exhaustiveness guards compared the two tables the test itself builds (require.Len(t, got, len(want))), which cannot catch a row deleted from both. Each table is now pinned to the literal count its package declares.
• 3898aca7, 4fcdf207: the header's survivor count and the S08 amendment count were both wrong; corrected to the measured six and to four.
• 5e169245: ociDefinedErrorCodeSet carried MANIFEST_UNVERIFIED (a Docker registry v2 code OCI never adopted) and omitted TOOMANYREQUESTS (which distribution-spec v1.1.1 defines), so a conformant registry answering 429 failed AC #42 (closed) on a correct response. Swapped, with the correction recorded under S07 §Resolutions.

Test plan

• go test ./... green.
• pre-commit run --all-files clean.
• Every new assertion verified in both directions. The error-code typo DIGEST_INVALID → DIGEST_INVALD kills both new OCI tests; "SIZE_INVALID" typo'd in the set, which the sweep could not reach, kills the set test; "upstream-url" → "upstream_url" kills the value table; duplicating a Field value kills both Field tests.

One thing the pass broke and had to repair

The first sweep run was stopped partway and left a file mutated with the package red — worse than not sweeping, since the next thing to run reports a failure with nothing to do with it. Restored, and the script now restores on SIGINT/SIGTERM/SIGHUP and at exit.

Recorded because it is the same lesson as the reference matrices: the tooling that produces evidence needs the same scrutiny as the code it measures, and this run has now hit that three times.