S31 Step 13: metadata merge and serve surface

Part of #290 (closed), S31 npm virtual. This slice composes code that is already merged: S11's hosted npm reads, S15's remote legs, and S13's resolution engine in internal/virtual, which Step 12 gave its first caller. The three npm_virtual_* tables land in internal/datastore/migrations/, the resolution reads in internal/datastore/, and the loading slice, the merge, the collection, and the serve surface in internal/format/npm/.

References: S31 spec · S31 plan (see "Step 13").

Type: feat | Depends on: Steps 8 (hosted document read seam), 12 (resolution facade and error mapping), and the shared internal/virtual collection mode (tracked as blocked-by links below).

The cross-upstream merge rewrote this step

!2018 (merged)'s decision, at note 3774142678, replaced the winner-only packument with a merge across every eligible upstream. The spec side landed with !2338 (merged), merged as c5ed5ef1d. #266 tracks the rest.

This step changes shape rather than being re-pointed. It was one MR serving one winner's document through a shared path; it becomes a three-MR stack that merges every contributor's document. The plan-side amendment travels as !2352 (merged): while that MR is open the plan on main still describes Step 13 winner-shaped, and once it merges the plan carries the three parts below. No MR for this step may open before !2352 (merged) merges, under the plan-before-code guardrail.

The word winner is now wrong for this step. It stays right for the tarball route, which Step 17 owns and the merge does not touch.

Two gates, and neither is Step 12

• The shared internal/virtual collection mode, which part 2 consumes and no step of this plan builds. Its plan travels as !2357 (merged); until that plan merges and its own steps land, internal/virtual offers no way to gather every position, only to elect one. This gates part 2 alone: parts 1 and 3 can be written beside it, because part 1 is pure and part 3 takes the merged stream rather than the positions.
• resolution.negative_cache_ttl and resolution.position_wait_timeout, two keys on S13's shared virtual_repositories.resolution block whose values are owed to the S13 DRI. Part 2 is where both bounds apply.

This step is what re-opens Step 12

#893 (closed) records four parts of Step 12 that re-open once the amendment lands. Part 2 below is what re-cuts them, not Step 12's own stack, which was deliberately built and reviewed against the spec as it stood. Of the four, the resolve_timeout derivation left Step 12 entirely when the S13 DRI moved the budget into the resolver (#1139 (closed)), the per-position Transform shipped per position already, Resolution acceptance criteria #11 (closed) and #12 turned out to be tarball-scoped and stay with Step 12, and the aggregate-verdict arms in virtual_read_errors.go are the one that lands here.

Scope

The cross-upstream merge for both metadata routes, the collection that feeds it, and the shared serve path for the merged document. Ships as three stacked MRs, in the order below, because each sits under the 500-line source ceiling while the whole step does not.

Part 1: the streaming merger

• internal/format/npm/virtual_packument_merge.go (Create). Pure: an ordered slice of per-position inputs, each a reader plus its position and upstream_repository_id, written to an io.Writer. No database, no HTTP, no context deadline of its own.

  versions streams and the tail buffers, and the split is not an optimization. Position 0's entries are emitted first and a later position's already-seen version key is never re-emitted, so versions never buffers a document that can run to tens of megabytes. dist-tags, time, and the top-level scalars are buffered and emitted after, because a value from the last position can still win one of them.

  The retained state is a version-key-to-digest-pair map, not a key set. A duplicate key is still read, because gitlab_artifact_registry_npm_virtual_integrity_divergence_total compares the candidates' dist.integrity and dist.shasum, and either can diverge alone.

• Acceptance: Spec AC, Packument merge #1 to #8 (closed), #10, #11 (closed), #21 (closed), and #23 (closed).

• Tests: one table per merge rule. The whole-object rule gets a case where the lower position's version object is missing a field the higher one has, asserting the field does not appear, since a field-by-field merge passes every test that only checks the fields both hold. time gets modified and created moving in opposite directions across three positions. Divergence gets a pair diverging on shasum alone and a pair diverging on integrity alone, which is what pins the digest pair. Plus the rapid state-machine property test over the position count and the input ordering, which the fixed-fixture determinism assertion cannot reach because it holds both still.

• Size: Source ~450 · Test ~950 · Total ~1,400

Part 2: collection over positions, and the outcome arms

• internal/format/npm/virtual_collect.go (Create). One input per eligible position over the foundation's collection mode, hosted through Step 8's seam and remote through the blob its cache row names.

  Both arms supply storage.WithForceStream(), and there are two call sites, not one. Without it on the remote OpenBlob, a redirect-configured namespace answers 307 straight to object storage and never reaches the merger. The WithForceStream() in internal/remote/singleflight.go does not cover this: it sits in readCommitted, the fill's own read-back.

  The three fan-out bounds land here. Concurrency is resolution.probe_concurrency, reused rather than renamed. The per-position wait is resolution.position_wait_timeout, on expiry of which the position is omitted rather than waited out. The negative cache for a position's 404 is in-process and per pod, under resolution.negative_cache_ttl, bounded by a fixed entry ceiling with LRU eviction because its key carries a client-supplied package name.

• internal/format/npm/virtual_read_errors.go (Modify): the aggregate verdict arms, re-cut from Step 12's winner vocabulary onto the merge's. merged and merged_partial are the metadata routes' only success values; the four winner-named values (served_from_cache, served_from_fallback, streamed, winner_fetch_failed) stay the tarball route's. hosted_rebuild_refused is the one winner-shaped value these routes do take.

  A shed hosted rebuild vetoes the read; it does not degrade it. A hosted position whose Lookup Hit is followed by a refused fenced rebuild fails the whole merge with S11's own code and Retry-After. This is the one arm of the collection that is not best-effort, and it is the arm most easily written the other way, since every neighboring failure is an omission.

• Acceptance: Spec AC, Packument merge #12 to #20 (closed) and #24 to #27 (closed). Plus four Spec AC, Resolution criteria cited by name: "A shed hosted rebuild vetoes the read rather than falling through to a remote" (wholly this part's, and the one Step 12's Acceptance bullet does not cite); "Nothing holds the package" (this part owns the metadata arm, Step 12 the tarball arm); and "A hosted miss alongside an unhealthy remote is a 404, not a 503" and "Outage is not an absence", which carry no route split and are asserted here through a collection where Step 12 asserts them through the facade. This part also picks up "A hosted package with no live version Misses both metadata reads", which no step owned; that gap predates the merge.

• Tests: the forced-stream pair once per position kind. The shed rebuild is asserted on the body, not on the remote receiving no request: collection asks every eligible position in parallel, so the remote is fetched and its contribution discarded, which is a different assertion from the tarball path's short-circuit. Each of the three refusal codes gets its own case. The negative cache gets the publish-hiding case as well as the hit case. The wait bound is asserted on the read finishing before the slow position's own timeout, not on the status alone, which a plain omission would also produce.

• Size: Source ~480 · Test ~1,050 · Total ~1,530

Part 3: the serve surface

• internal/format/npm/virtual_document_serve.go (Create). Reuses the five plain-argument helpers from remote_document_serve.go and grows parallel equivalents for the five that take remote.ServeResult. serveRemoteUpstream's parallel is the one an inventory stopping at three misses.

  The serve-time re-point applies to every emitted entry, whichever position it came from. A remote contributor's document already carries its own repository's URLs, because the fill-time transform is built per position on that base by Step 11's wrapper; this step applies the rewriter a second time on the virtual base. A hosted contributor's entries are built with the virtual base directly.

  The ETag is derived from the merge inputs, not from the served bytes. An algorithm version byte, then per position its position, upstream_repository_id, and that input's blob SHA-256, so If-None-Match answers 304 after the database reads alone. A HEAD is not that cheap: it runs the collection and the merge and discards the body, because an over-cap or unparseable input is only discovered in the parse.

  Hosted rebuildSem pressure rises under the merge rather than moving: every eligible hosted position contributes on every read, where the winner-only path reached at most one.

• Acceptance: Spec AC, Packument re-point #1 to #7 (closed), and #8 (closed) as the primary assertion on both metadata routes (Cache-Control comes from the virtual repository's visibility, not a contributor's). Spec AC, Conditional requests #1 to #5. Plus Spec AC, Packument merge #9, #22 (closed), and #28 (closed).

• Tests: a re-point assertion that every emitted entry carries the virtual base, since a re-point applied to position 0's entries alone passes any assertion checking the first one. A 304 case asserting zero blob reads, driven by a counting store. The HEAD case runs on a fixture whose last contributor fails to parse, where the GET answers 503; a fixture where every input parses passes whether or not the HEAD runs the merge.

• Size: Source ~400 · Test ~950 · Total ~1,350

Size

Whole step: Source ~1,330 · Test ~2,950 · Total ~4,280. Each part is under the 500-line source ceiling; only the sum is over, which is why this is a stack rather than a step with an apology.