feat(npm): hosted tarball lookup and its Phase 1 upstream adapter (S31 plan: 5/19)

What

Completes plan step 5 on top of the metadata lookups: the third batched authoritative read, for tarballs, and the remote.Upstream adapter that presents all three document kinds to the virtual resolver as Phase 1 answers.

Stacked MRs

Part	MR	Scope	Reviewable LOC	Target
1/2	!1974 (merged)	the two metadata reads	1,091	main
2/2	this MR	tarball read and the Phase 1 upstream adapter	1,234	main

Merge order is 1 then 2, and it is not optional. This MR's VirtualHostedLookupStore seam declares one method per document kind, so it cannot compile before part 1's reads exist. Part 1 has merged, so this MR is rebased onto main and targets it directly.

Part 1 also carries a deliberate departure from the spec on the dist-tags predicate that needs a ruling. It is described in part 1 and is not re-argued here.

The tarball read

It is the one projecting a column beyond the position id, npm_files.blob_sha256, because only a tarball hit names a committed blob.

unique_npm_files_ns_id_version_id_file_name is unique per version and not per package, so nothing in the schema stops one file name sitting under two live versions of one package and matching twice for one position. The npm publish path does not write that shape, since it derives the file name from the version, so DISTINCT ON the position is defence in depth rather than a fix for a row shape in production today. Ordering by created_at DESC then id DESC resolves a match to the current tarball rather than to a resurrected predecessor; npm_files.id is the leading column of the composite primary key, which the bound namespace_id makes unique here.

Three properties no result set shows

• The DISTINCT ON key must be the column the IN list binds, or the bound becomes that column's cardinality, which grows with the table. Asserted on the rendered SQL.
• The whole ORDER BY, whose trailing unique column is what makes the surviving row deterministic between runs. Asserted on the rendered SQL.
• The join direction. The read binds file_name but no version, so driven from npm_packages it would walk every live version of the package and probe npm_files once each, and ADR-004 permits 25,000 of them. Asserted on an EXPLAIN ANALYZE plan, against a fixture seeded with enough decoy versions to make the rejected plan the visibly worse one.

The assertion is the direction rather than the index npm_files is reached by: which index wins is data-dependent, and on a fixture this size PostgreSQL filters file_name instead of binding index_npm_files_on_ns_id_file_name.

The adapter

A hosted Hit is always Fresh, because the read tests the authoritative rows and has no cache to be stale against, so a hosted position never enters Phase 2. Only a tarball Hit carries a blob reference. An invalid target errors rather than reading as a Miss; a blob_sha256 of the wrong length is reported rather than rendered into an address naming no blob; and a hit for a position outside the requested batch is refused rather than added to the answer. Each position keeps the npm_repositories.id its answer came from, which remote.Upstream has no field for, so the resolver can name the position it consulted.

remote.LookupResult.BlobRef may now be empty on a Hit

A hosted upstream generates a metadata document rather than caching one, so this MR is the first thing in the tree to falsify the field's old promise. The doc now says the reference is optional without prescribing a discriminator: a remote (kind=2) metadata Hit does carry a reference, so the requested document kind alone does not decide, and the winner's upstream kind is what does.

remote.CacheStore.Lookup states the invariant Standalone.openCached turns into a blob address without a kind check, and internal/virtual's own restatements of the promise (ResolveServeFromCache, ResolveResult.Cache) are widened to match, so the contract does not ship half-changed.

Diff size

1,234 reviewable LOC, over the 500 LOC guidance in development-model.md, so here is the split and the reason:

Group	LOC
Production Go	305
Tests	872
Docs	57
Total	1,234

Production code is 305 LOC, inside the ceiling. Step 5 was already split into two MRs of roughly equal size to keep each reviewable, and this part is the smaller half of the production code.

This part deliberately carries two things, the tarball read and the adapter over all three kinds, and that is the accepted trade-off rather than an oversight. The adapter's seam declares one method per document kind, so it cannot compile before the tarball read exists; splitting it out into a third MR would leave a part whose only content is an interface no caller reaches.

Testing

• The tarball read: positive hit carrying the blob, a Miss at every soft-delete level (file, version, package), cross-namespace and unlisted positions, the empty batch, statement failure, and the empty-file-name guard. The read also goes through part 1's shared argument validator, so part 1's zero-position rejection now runs once per document kind, tarballs included.
• The DISTINCT ON collapse driven by the only row shape that changes the answer: one file name under two live versions, seeded in reverse order so the tiebreaker decides.
• The adapter: Fresh always true on a Hit, a blob reference for tarballs alone, the zero values of both exported types, the nil-store panic, error propagation, a short blob_sha256 refused, an invalid target refused with the sentinel Target.Validate actually returns, and a hit outside the requested batch refused.
• One statement per document kind at 2 and at 20 positions, now including tarballs.

Integration suite run locally against a CI-tuned PostgreSQL; golangci-lint run with --build-tags=integration, which CI lint cannot see.

e2e catalog

No scenario added or affected. This step adds no route and no handler: nothing reaches this code until the composition root in step 14, and the virtual e2e scenarios are step 19's deliverable.

Closes #886 (closed)