S31 Step 5: hosted upstream lookup

Part of #290, S31 npm virtual. This slice composes code that is already merged: S11's hosted npm reads, S15's remote legs, and S13's resolution engine in internal/virtual, which has had no caller until now. The three npm_virtual_* tables land in internal/datastore/migrations/, the resolution reads in internal/datastore/, and the loading slice, the serve surface, and the three read handlers in internal/format/npm/.

References: S31 spec · S31 plan (see "Step 5").

Type: feat | Depends on: nothing. This step is a root and can start immediately.

Scope

The batched authoritative reads for a hosted position and the remote.Upstream adapter over them, reporting a Hit with no blob reference.

Files

  • internal/datastore/npm_virtual_hosted_lookup.go (Create): one batched IN-list read per document kind over the authoritative tables, keyed on the npm_repositories.id Step 4's third join supplies. npm_packages joined to npm_versions for a packument, npm_tags for a dist-tags document, and npm_files for a tarball. Three of those four tables carry soft_deleted_at, and every read filters soft_deleted_at IS NULL on the ones that do. npm_tags is the exception and carries no such column: tags are hard-deleted, and the whole-package soft-delete is enforced through the package row, so the dist-tags read reaches its soft-delete predicate through the join, filtering npm_packages.soft_deleted_at IS NULL and the bound npm_versions.soft_deleted_at IS NULL. Without the predicate an unpublished package, version, or tarball is reported as a hosted Hit and served through the virtual route to a wider audience than the hosted repository's own.

    Each of the three reads is a Hit-or-Miss question plus, for the tarball kind alone, one blob reference. None of them needs the matched rows, so the packument and dist-tags reads are written as EXISTS(...) probes rather than row-returning joins. That matters for the packument: npm_packages joined to npm_versions returns one row per version, and ADR-004 permits 25,000 versions per package, so a row-returning form is an unbounded read meeting none of the project's SELECT-bound exemptions. The IN list on all three is the hosted-position count, which shares the list read's ADR-004 bound; Step 4's LIMIT is what makes that true, and this step states the dependency rather than assuming it. Not npm_metadata_files: S11 force-expires those rows inside every write transaction, so a cache-scoped Lookup reports a Miss for a package the upstream demonstrably holds, and on a list like [internal-hosted, npmjs-remote] that Miss is a package-substitution hazard.

  • internal/format/npm/virtual_hosted_upstream.go (Create): the remote.Upstream adapter. A tarball Hit carries npm_files.blob_sha256 as its blob reference; a packument or dist-tags Hit carries none, because no blob is guaranteed to exist. Fresh is always true: a hosted Lookup is authoritative and never stale, so a hosted position never enters Phase 2.

  • internal/remote/operations.go (Modify): LookupResult.BlobRef's doc comment says it is "the content-addressable reference of the cached blob when Hit is true", which this adapter is the first thing in the tree to falsify. The comment gains the hosted-upstream case, so the next slice reading Cache.BlobRef without a winner-kind check is warned by the type rather than by a nil digest reaching OpenBlob.

  • internal/datastore/query_names.go (Modify): the hosted-lookup query names, in their own group.

  • the paired *_integration_test.go and *_test.go files (Create).

Acceptance

The store-level half of Spec AC, Resolution #5: a hosted Lookup reports a Hit for a package whose npm_metadata_files rows are all force-expired, for all three document kinds. Contributes the hosted half of #15: one statement per document kind regardless of hosted position count.

Tests

Integration for each of the three document kinds; the force-expired-cache case asserted explicitly; a soft-deleted row asserted as a Miss, once per document kind, which is the case that decides whether unpublished content is reachable through a virtual route. The dist-tags case seeds a soft-deleted npm_packages row (and separately a soft-deleted npm_versions row) with the npm_tags row left in place, since npm_tags has no soft_deleted_at to set: that is the shape the predicate has to catch, and a test author who reaches for a soft-deleted tag row will find no column and quietly drop the case. Plus a package absent from every hosted position; the batched statement count; and a plan-shape assertion that the packument probe does not fan out over versions. Unit tests for the adapter's Fresh and blob-reference contract, including the zero-value behavior of the result it returns.

Size

Source ~440 · Test ~1,000 · Total ~1,440

🤖 Triage note: This issue has been reviewed by the triage assistant and classified as type::feature — it introduces new functionality (hosted tarball lookup and a Phase 1 upstream adapter for npm virtual repositories) rather than fixing a defect or performing maintenance work. The labels Category:Artifact Registry, devops::package, and group::package registry are already applied.

If this classification doesn't look right to you, feel free to update the labels and let us know!

Edited by Triage and Label issues