S31 Step 11: upstream assembly and memoized lookups

Part of #290, S31 npm virtual. This slice composes code that is already merged: S11's hosted npm reads, S15's remote legs, and S13's resolution engine in internal/virtual, which has had no caller until now. The three npm_virtual_* tables land in internal/datastore/migrations/, the resolution reads in internal/datastore/, and the loading slice, the serve surface, and the three read handlers in internal/format/npm/.

References: S31 spec · S31 plan (see "Step 11").

Type: feat | Depends on: Steps 4, 5, 6, and 7 (tracked as blocked-by links below).

Scope

The loading slice. Turn a virtual repository and a target into []virtual.UpstreamSpec plus a virtual.Coordinate, with every Lookup batched and memoized, every health verdict prefetched, and every remote position's flight resolved lazily.

Files

Injected seams: the assembly is constructed with npmremote.FlightRegistry (the only source of the shared per-repository *remote.SingleFlight that RemoteUpstreamParams.Fetcher requires) and npmremote.CacheStores (the source of each position's remote.CacheStore). Neither is produced by Steps 4 to 7, and virtual.NewRemoteUpstream panics without them, so they are constructor parameters here and Step 14 supplies the concrete values. Naming them in this step is the whole point: they are the shared machinery that, left unowned, is what three parallel S15 stacks each rebuilt.

  • internal/format/npm/virtual_assembly.go (Create): load the list and the rules, run the two batched Lookups, classify each row's kind, and build one UpstreamSpec per position through virtual.NewHostedUpstream, virtual.NewRemoteUpstream, or virtual.NewUnexpectedKindUpstream. The jet-row-to-values adaptation for the rules lives here, not in Step 7. It also keeps the index-to-stored-position mapping alongside the loaded rows, because ResolveResult.Position is an index into the slice the assembly built and the position column is unique but not contiguous, so the two disagree for exactly the lists an operator is debugging. Step 18 reads the mapping rather than the index.

    An out-of-range kind is classified and handed over, never dropped: the foundation is what records the exclusion under a data-error reason, and a dropped row would be invisible to the audit stream.

    A kind=2 position whose npm_remote_repositories row is absent is the sibling of the out-of-range kind and gets its own constructor rather than the unexpected-kind one. Step 4's list read LEFT JOINs that table, so the row can be missing, and virtual.NewRemoteUpstream panics on the resulting nil Cache. virtual.NewUnexpectedKindUpstream cannot take it either, because kind=2 is in range and that constructor panics on an in-range kind by contract. Fabricating an out-of-range kind to slip past that panic is worse than both: the audit stream would then record unexpected_kind for a row whose kind is genuinely remote.

  • internal/format/npm/virtual_memo.go (Create): three per-request adapters, each closing one gap between a foundation seam and the fixed statement count.

    • The memoizing remote.Upstream over the prefetched Lookup map, one per position. This is what reconciles Phase 1's per-position Lookup with the spec's enumeration.

    • A memoizing remote.HealthStatusReader that answers from a per-request map carried on the context and falls back to the store on a miss. The map is filled from the last_health_status column Step 4's join already carries. The fallback branch increments a counter and logs at debug with the repository id, because a path that forgets to populate the map is otherwise a silent return to one statement per position. A NULL last_health_status is not a map miss: a miss falls through to the store, which answers ErrNotFound, which unhealthyRemote turns into an aborted resolution and a 500 for every read of the repository. So the map is filled with an explicit 0 (unknown) for a NULL and is keyed on presence rather than on a zero value.

    • A lazy fetcher that resolves its position's flight from the FlightRegistry on first Fetch and caches it for the request, so a denied or skipped position never takes the registry's process-wide lock. It does not shorten any credential's lifetime, and this step does not claim it does; the control that does is Step 4's projection.

      This wrapper also builds the fill-time rewriter, one per position. virtual.Request.Fetch is request-global, so a rewriter built once for the request carries whichever position's base it was built on into every position's fill, and this wrapper is the innermost place a position's own identity is still in scope. The base is packumentBaseURL(publicRegistryURL, <request namespace slug>, repositories.name), byte-identical to the standalone route's. Step 12 owns the BodyKind, the size cap, and the npm.public_registry_url unset guard.

  • internal/virtual/upstream.go (Modify): a fourth position constructor, NewMissingRemoteBindingUpstream(repositoryID uuid.UUID, kind UpstreamKind, rules []Rule) UpstreamSpec. It keeps the observed kind rather than a fabricated out-of-range one, fails the position closed exactly as an unexpected-kind position does, carries rules so the classification can outrank them, and panics on a zero repositoryID. The mark it sets has to be distinguishable from the unexpected-kind one, because EligibleUpstreams currently classifies on the presence of that mark alone.

  • internal/remote/audit.go (Modify): ExclusionRemoteBindingMissing, the parallel UpstreamExclusionReason with token remote_binding_missing, appended after ExclusionUnexpectedKind.

  • internal/virtual/rules.go (Modify): ReasonRemoteBindingMissing, classified from the missing-binding mark before rules are consulted, exactly as ReasonUnexpectedKind already outranks them, and counted as a data error in the same arm so the aggregate stays ErrAllUpstreamsIneligible.

  • internal/virtual/resolve.go (Modify): a loadAndFilter case routing ReasonRemoteBindingMissing to remote.ExclusionRemoteBindingMissing, failing the position closed and setting the data-error flag like the sibling arms, with its own ERROR log distinct from logUnexpectedKind.

    These four bullets are this plan's one accepted S13 foundation touch, and it is additive. No existing signature, panic condition, or enum value moves, so nothing already merged behaves differently and no step is gated behind it; all four edits land in this step's own MR. A spec amendment rides behind the new audit reason and gates nothing.

  • internal/format/npm/virtual_assembly_test.go, virtual_memo_test.go, virtual_assembly_property_test.go, and the paired internal/virtual/upstream_test.go, internal/virtual/rules_test.go, internal/virtual/resolve_test.go, and internal/remote/audit_test.go cases (Create and Modify).

Acceptance

Spec AC, Resolution #13 (closed) (an upstream of an unexpected kind is ineligible, error-logged, recorded under a data-error reason, and not recursed into), #14 (a soft-deleted upstream is skipped rather than counted as a failure, so a lower-priority upstream still serves), #15 end to end, and #16 (closed) (the health status costs no statement). The last two move here from Step 4 because they are properties of a composed resolution and a store-level count cannot fail on them.

Tests

The winner rule gets a rapid state-machine property test over position count, per-position outcome, and rule eligibility, asserting the lowest-position-success invariant; docs/dev/go-testing.md's ## Technique selection names virtual registry upstream resolution order as a property-testing default.

Everything else is table-driven: kind classification including the out-of-range row and the kind=2 row with no npm_remote_repositories binding, each asserted to produce an audited exclusion and not a panic, and each under its own reason (unexpected_kind for the first and remote_binding_missing for the second) so a run that routed both through one constructor fails. The four foundation edits carry their own cases: NewMissingRemoteBindingUpstream accepts an in-range kind, reports that observed kind from Kind(), and panics on a zero repositoryID; EligibleUpstreams classifies such a position ReasonRemoteBindingMissing and not ReasonUnexpectedKind, whatever rules it carries, and still counts as a data error in the aggregate; loadAndFilter audits it under remote.ExclusionRemoteBindingMissing, logs it through its own helper, and leaves it never looked up, probed, fetched, or recursed into; and the new reason's String() token is asserted alongside its siblings.

Plus the soft-delete skip; a memo test proving a position queried twice issues one statement; a health-memo test proving 20 eligible remote positions issue no health statement; a NULL last_health_status asserted to leave its position eligible rather than aborting the resolution, with the store asserted untouched; a fallback-counter assertion that an unpopulated map is counted rather than silently absorbed; an end-to-end statement count over a real 2-upstream and 20-upstream resolution asserting the same total; a test that a denied position's flight is never resolved; and a test that the index-to-position mapping survives a non-contiguous list such as [0, 2, 5].

Size

Source ~575 · Test ~1,290 · Total ~1,865. At ~575 source this is the largest non-schema step and it is over the plan's ceiling, so the MR description justifies it with the split by file group.