docs(specs): add the S31 npm virtual repositories spec

What

Adds docs/specs/S31-npm-virtual.md, the spec for npm virtual repositories (kind=1), and links it from the spec index.

A virtual repository composes hosted (S11) and remote (S15) npm repositories into one ordered upstream list and resolves each request through S13's single-winner algorithm. The spec owns the npm-specific pieces on top of that: repository-kind dispatch, the virtual schema, the npm allow/deny target fields, the dist.tarball re-point, error mapping to the npm envelope, and the read-only client surface.

Key decisions

  • Single winner, no merge. The document served is the winner's, not a union across upstreams. Cross-upstream version union and dist-tag reconciliation is deferred to #266.
  • The dist.tarball re-point runs on the serve path. S15's streaming rewrite is applied a second time, with the virtual repository as the base URL, so a client following a served packument stays inside the repository it addressed. The remote cache holds the upstream's own document rather than per-version rows, and the winner's cached document is never rewritten in place: it is shared with direct reads of that repository and with every other virtual repository listing it.
  • No cache and no credentials of its own. Every document and tarball a virtual repository serves lives in its upstreams' tables. The three new tables hold only the composition.
  • Tarball bytes are relayed, not verified. They stream through S13's teed Fetch, and dist.shasum and dist.integrity are relayed byte-for-byte, so the client's own check is the gate. There is nothing for the virtual layer to verify or withhold.
  • ETag is the winner's cached blob digest, and a cold streamed fill carries none. Cache-Control reads the virtual repository's visibility, not the winner's.
  • Variant follows Accept, with no cross-variant derivation. Each variant is fetched and cached independently on both upstream kinds.
  • No configuration. Resolution and health knobs come from S13's virtual_repositories; the rewrite base and max_remote_packument_size (64MB) come from S11's NpmConfig, which the remote path already passes per request.

Resolved in review (ba5ec835)

  • The merged S17 phase 6 overtook the review. The dependency row now records the merged spec, the delete-guard merge order points at the guard's implementation (tracked in #314), and the sites saying the association surface is "not specified yet" now say "not implemented yet". The merged guard answers 409, code conflict — shared with repository-not-empty — with a static message naming the in-use condition, and the spec records the message as the distinguisher.
  • !1502 (merged) merges before this spec too, not only before the rule-evaluation step MR: S31 quotes the amendment's heading and attributes its scoping and invalid-rule verdicts to S13, and today's S13 states the opposite of the eligibility rule.
  • The same-namespace upstream constraint is ADR-007's (applying ADR-001's namespace scoping), not ADR-022's, at all three sites; the S13 copy of the same gloss gets fixed in !1502 (merged).
  • WinnerFetchFailed relays the upstream's own Retry-After where the response carried one, per the amendment's relay rule, and the error-cases text now lists which rows carry which value instead of "not on every 503".
  • A virtual tarball takes the metadata Cache-Control set, not S11's max-age=31536000, immutable: the winner can change between reads, so the same URL serves different bytes over time, and immutable would let a shared cache mask exactly the change the winner-change criterion keeps visible.
  • HEAD on the packument and dist-tags routes is S31's own extension over S11, which grants HEAD on the tarball alone; stated in API Contracts so a hosted-side 405 on those routes cannot silently break this spec.

Resolved in review (8d22cc8f)

  • An all-ineligible read answers 500, not 404. A corrupt rule or an upstream of an unexpected kind is not a verdict about the artifact, so folding it into the not-found arm reports an absence nothing established — and npm caches a 404 as a hard E404 it never retries. This needs the foundation to keep the two aggregates apart, since the resolver collapses them today, so !1502 (merged) grew a third amendment: AllUpstreamsIneligible next to AllUpstreamsDenied, plus upstream-kind eligibility as a resolution-time concern (S13's standing text said the opposite).
  • HeaderInjection answers 500 rather than a 503 promising a retry that cannot change anything, and a single-flight timeout keeps its 503 without Retry-After. The header is on the two verdicts !1502 (merged) fixes it for.
  • The upstream-delete guard bites at the delete request. NO ACTION alone would leave S20-A's purger stalling on 23503 forever, with no operator signal, so S17's repository delete refuses a repository still listed as an upstream. Recorded in Dependencies with a merge order.
  • The legacy GET /{package}/{tag} form is out of scope, answering 404: S11 defines no such route, so a hosted winner cannot serve it and a client's success would depend on which kind won.
  • Authorization precedes the kind dispatch, so a refused write does not confirm that a private repository exists and is virtual.
  • Smaller fixes in the same commit: the tarball ETag takes the cold-fill carve-out, If-Modified-Since is answered only where Last-Modified was emitted, Content-Length is omitted rather than optionally set from the transformed bytes, the re-point derivation names {name}, the health status is shown to come off the upstream-list join, and the wide event gains no_upstreams and single_flight_timeout.

Resolved in the first revision

  • target_field scoping belongs to the foundation. The npm slice supplies its three target fields and the values the request carries, and S13 decides which rules that leaves in play; S31 filters nothing. S32 (!972 (merged)) reached the same requirement for tag-targeted rules on a digest request, which is what places it below both slices. The S13 amendment is !1502 (merged), and the foundation change it implies — including the internal/virtual matcher work that lets a corrupt target_field stay distinguishable from an unaddressed one — is tracked in #609 (closed).
  • Retry-After takes S13's value: health_check.scheduled_interval in whole seconds, 300 by default. Also !1502 (merged), and now scoped there to the verdicts that describe upstream health.
  • Rule CRUD is not in the closed-beta management surface. S17 phase 6 (!1408 (merged)) settles it for every virtual slice: associations ship rule-less, so npm_virtual_upstream_rules stays reachable only by direct SQL until a rule-CRUD patch lands, and ADR-009 is amended alongside those routes.

Open questions

  • Whether an upstream may be less visible than the virtual repository that lists it. Whether S17's association endpoint should refuse the pairing, treat it as a privileged action, or allow it. For the S17 and S13 DRIs together, because the answer applies to every virtual slice. S31 needs no change whichever way it lands: the read path behaves identically and only the write-side guard moves.

Notes

  • Status: Approved in the spec file, per the spec-MR guardrail; merging is the approval signal.
  • Implementation depends on S15's remote operations and proxy handlers (steps 9-12) landing first, because a remote-kind upstream's Lookup, Probe, and Fetch are exactly those operations. The spec's Dependencies section records it.
  • Merge order: !1502 (merged) lands before this spec and before the S31 step MR that implements rule evaluation. The spec's Depends on line records both, because the step MR would otherwise carry the filter this revision drops, and the spec itself cites the amendment's text as S13's.
  • Merge order: the S17 repository-delete guard's implementation (specified in the merged S17, tracked in #314) lands before the S31 schema MR that creates npm_virtual_repository_upstreams. The stall starts with the first association row, so the two cannot be separated by a release.
  • Deferred follow-ups tracked in #266. AppSec review for the format tracked in #244.

Related to #285 (closed)

Edited by Dzmitry (Dima) Meshcharakou

Merge request reports

Loading
Loading