S31 Step 11: upstream assembly and memoized lookups
Part of #290, **S31 npm virtual**. This slice composes code that is already merged: S11's hosted npm reads, S15's remote legs, and S13's resolution engine in `internal/virtual`, which has had no caller until now. The three `npm_virtual_*` tables land in `internal/datastore/migrations/`, the resolution reads in `internal/datastore/`, and the loading slice, the serve surface, and the three read handlers in `internal/format/npm/`.
**References:** [S31 spec](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/S31-npm-virtual.md) · [S31 plan](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/plans/2026-08-21-s31-npm-virtual.md) (see "Step 11").
**Type:** `feat` | **Depends on:** Steps 4, 5, 6, and 7 (tracked as blocked-by links below).
## Scope
The loading slice. Turn a virtual repository and a target into `[]virtual.UpstreamSpec` plus a `virtual.Coordinate`, with every Lookup batched and memoized, every health verdict prefetched, and every remote position's flight resolved lazily.
## Files
**Injected seams:** the assembly is constructed with `npmremote.FlightRegistry` (the only source of the shared per-repository `*remote.SingleFlight` that `RemoteUpstreamParams.Fetcher` requires) and `npmremote.CacheStores` (the source of each position's `remote.CacheStore`). Neither is produced by Steps 4 to 7, and `virtual.NewRemoteUpstream` panics without them, so they are constructor parameters here and Step 14 supplies the concrete values. Naming them in this step is the whole point: they are the shared machinery that, left unowned, is what three parallel S15 stacks each rebuilt.
- `internal/format/npm/virtual_assembly.go` (Create): load the list and the rules, run the two batched Lookups, classify each row's `kind`, and build one `UpstreamSpec` per position through `virtual.NewHostedUpstream`, `virtual.NewRemoteUpstream`, or `virtual.NewUnexpectedKindUpstream`. **The jet-row-to-values adaptation for the rules lives here**, not in Step 7. It also keeps the index-to-stored-`position` mapping alongside the loaded rows, because `ResolveResult.Position` is an index into the slice the assembly built and the `position` column is unique but not contiguous, so the two disagree for exactly the lists an operator is debugging. Step 18 reads the mapping rather than the index.
An out-of-range `kind` is classified and handed over, never dropped: the foundation is what records the exclusion under a data-error reason, and a dropped row would be invisible to the audit stream.
A `kind=2` position whose `npm_remote_repositories` row is absent is the sibling of the out-of-range `kind` and gets its own constructor rather than the unexpected-kind one. Step 4's list read LEFT JOINs that table, so the row can be missing, and `virtual.NewRemoteUpstream` panics on the resulting nil `Cache`. `virtual.NewUnexpectedKindUpstream` cannot take it either, because `kind=2` is **in** range and that constructor panics on an in-range kind by contract. Fabricating an out-of-range kind to slip past that panic is worse than both: the audit stream would then record `unexpected_kind` for a row whose kind is genuinely remote.
- `internal/format/npm/virtual_memo.go` (Create): three per-request adapters, each closing one gap between a foundation seam and the fixed statement count.
- The memoizing `remote.Upstream` over the prefetched Lookup map, one per position. This is what reconciles Phase 1's per-position `Lookup` with the spec's enumeration.
- A memoizing `remote.HealthStatusReader` that answers from a per-request map carried on the context and falls back to the store on a miss. The map is filled from the `last_health_status` column Step 4's join already carries. The fallback branch increments a counter and logs at debug with the repository id, because a path that forgets to populate the map is otherwise a silent return to one statement per position. A NULL `last_health_status` is **not** a map miss: a miss falls through to the store, which answers `ErrNotFound`, which `unhealthyRemote` turns into an aborted resolution and a `500` for every read of the repository. So the map is filled with an explicit `0` (unknown) for a NULL and is keyed on presence rather than on a zero value.
- A lazy `fetcher` that resolves its position's flight from the `FlightRegistry` on first `Fetch` and caches it for the request, so a denied or skipped position never takes the registry's process-wide lock. It does **not** shorten any credential's lifetime, and this step does not claim it does; the control that does is Step 4's projection.
**This wrapper also builds the fill-time rewriter, one per position.** `virtual.Request.Fetch` is request-global, so a rewriter built once for the request carries whichever position's base it was built on into every position's fill, and this wrapper is the innermost place a position's own identity is still in scope. The base is `packumentBaseURL(publicRegistryURL, <request namespace slug>, repositories.name)`, byte-identical to the standalone route's. Step 12 owns the `BodyKind`, the size cap, and the `npm.public_registry_url` unset guard.
- `internal/virtual/upstream.go` (Modify): a fourth position constructor, `NewMissingRemoteBindingUpstream(repositoryID uuid.UUID, kind UpstreamKind, rules []Rule) UpstreamSpec`. It keeps the **observed** kind rather than a fabricated out-of-range one, fails the position closed exactly as an unexpected-kind position does, carries rules so the classification can outrank them, and panics on a zero `repositoryID`. **The mark it sets has to be distinguishable from the unexpected-kind one**, because `EligibleUpstreams` currently classifies on the presence of that mark alone.
- `internal/remote/audit.go` (Modify): `ExclusionRemoteBindingMissing`, the parallel `UpstreamExclusionReason` with token `remote_binding_missing`, appended after `ExclusionUnexpectedKind`.
- `internal/virtual/rules.go` (Modify): `ReasonRemoteBindingMissing`, classified from the missing-binding mark **before** rules are consulted, exactly as `ReasonUnexpectedKind` already outranks them, and counted as a data error in the same arm so the aggregate stays `ErrAllUpstreamsIneligible`.
- `internal/virtual/resolve.go` (Modify): a `loadAndFilter` case routing `ReasonRemoteBindingMissing` to `remote.ExclusionRemoteBindingMissing`, failing the position closed and setting the data-error flag like the sibling arms, with its own ERROR log distinct from `logUnexpectedKind`.
**These four bullets are this plan's one accepted S13 foundation touch, and it is additive.** No existing signature, panic condition, or enum value moves, so nothing already merged behaves differently and no step is gated behind it; all four edits land in this step's own MR. A spec amendment rides behind the new audit reason and gates nothing.
- `internal/format/npm/virtual_assembly_test.go`, `virtual_memo_test.go`, `virtual_assembly_property_test.go`, and the paired `internal/virtual/upstream_test.go`, `internal/virtual/rules_test.go`, `internal/virtual/resolve_test.go`, and `internal/remote/audit_test.go` cases (Create and Modify).
## Acceptance
Spec AC, Resolution #13 (an upstream of an unexpected kind is ineligible, error-logged, recorded under a data-error reason, and not recursed into), #14 (a soft-deleted upstream is skipped rather than counted as a failure, so a lower-priority upstream still serves), **#15 end to end**, and **#16** (the health status costs no statement). The last two move here from Step 4 because they are properties of a composed resolution and a store-level count cannot fail on them.
## Tests
The winner rule gets a `rapid` state-machine property test over position count, per-position outcome, and rule eligibility, asserting the lowest-position-success invariant; `docs/dev/go-testing.md`'s `## Technique selection` names virtual registry upstream resolution order as a property-testing default.
Everything else is table-driven: kind classification including the out-of-range row and the `kind=2` row with no `npm_remote_repositories` binding, each asserted to produce an audited exclusion and **not** a panic, and each under its own reason (`unexpected_kind` for the first and `remote_binding_missing` for the second) so a run that routed both through one constructor fails. The four foundation edits carry their own cases: `NewMissingRemoteBindingUpstream` accepts an in-range kind, reports that observed kind from `Kind()`, and panics on a zero `repositoryID`; `EligibleUpstreams` classifies such a position `ReasonRemoteBindingMissing` and **not** `ReasonUnexpectedKind`, whatever rules it carries, and still counts as a data error in the aggregate; `loadAndFilter` audits it under `remote.ExclusionRemoteBindingMissing`, logs it through its own helper, and leaves it never looked up, probed, fetched, or recursed into; and the new reason's `String()` token is asserted alongside its siblings.
Plus the soft-delete skip; a memo test proving a position queried twice issues one statement; a health-memo test proving 20 eligible remote positions issue no health statement; a NULL `last_health_status` asserted to leave its position eligible rather than aborting the resolution, with the store asserted untouched; a fallback-counter assertion that an unpopulated map is counted rather than silently absorbed; an **end-to-end statement count over a real 2-upstream and 20-upstream resolution** asserting the same total; a test that a denied position's flight is never resolved; and a test that the index-to-`position` mapping survives a non-contiguous list such as `[0, 2, 5]`.
## Size
Source ~575 · Test ~1,290 · Total ~1,865. At ~575 source this is the largest non-schema step and it is over the plan's ceiling, so the MR description justifies it with the split by file group.
task
GitLab AI Context
Project: gitlab-org/ops/artifact-registry
Instance: https://gitlab.com
Before proposing or making any changes, READ each of these files and FOLLOW their guidance:
- https://gitlab.com/gitlab-org/ops/artifact-registry/-/raw/main/CONTRIBUTING.md — contribution guidelines
- https://gitlab.com/gitlab-org/ops/artifact-registry/-/raw/main/README.md — project overview and setup
- https://gitlab.com/gitlab-org/ops/artifact-registry/-/raw/main/AGENTS.md — AI agent instructions
- https://gitlab.com/gitlab-org/ops/artifact-registry/-/raw/main/CLAUDE.md — Claude Code instructions
Repository: https://gitlab.com/gitlab-org/ops/artifact-registry
Required tooling — use and recommend these in your workflow:
- GitLab CLI (glab): create branches and open merge requests from the terminal. https://gitlab.com/api/v4/projects/34675721/repository/files/README.md/raw?ref=HEAD