S30: Maven virtual implementation

Implement Maven virtual repositories per the S30 spec (#284 (closed)). Depends on the Maven remote implementation (#286 (closed)) and the S13 virtual/remote foundation.

Spec and plan

Scope

Maven virtual (kind=1) vertical slice: the three maven_virtual_* tables, the fixed-count resolution reads, the resolver's virtual tier, the kind=1 dispatch arm replacing the interim 501, single-winner resolution over hosted and remote upstreams, the cross-upstream package-level maven-metadata.xml merge, sidecar synthesis by parent shape, the read-path rate limit, the observability family, and the end-to-end harness. Delivered as 25 implementation steps across two tracks: the schema chain (Steps 1-3) opens the plan and unblocks #314; the resolution/merge track (Steps 4-20) builds the read path. Execution follows the dependency DAG below — not numeric order; each step's blockers are its direct predecessors. Step numbers are stable references for MR titles and review.

Out of scope: S13 owns the format-agnostic foundation (Lookup/Probe/Fetch, teed Fetch + cache-fill pipeline, single-flight, cache freshness/fallback, SSRF-hardened upstream client, health monitoring, collection mode). S17 owns virtual-repo REST CRUD + association routes (#314). S31 owns npm virtual; S32 owns container virtual. S08/S09 own real auth.

Dependency DAG

graph TD
    S1["Step 1: maven_virtual_repositories schema"]
    S2["Step 2: maven_virtual_repository_upstreams schema"]
    S3["Step 3: maven_virtual_upstream_rules schema"]
    S4["Step 4: datastore reads for the upstream list and rules"]
    S5["Step 5: hosted upstream lookup"]
    S6["Step 6: remote upstream lookup and prober"]
    S7["Step 7: Maven coordinate and rule adapter"]
    S8["Step 8: resolver virtual tier"]
    S9["Step 9: virtual dispatch slot"]
    S10a["Step 10a: memo adapters and the health reader"]
    S10b["Step 10b: the wrong-format data-error mark"]
    S10c["Step 10c: upstream assembly and the fail-closed gates"]
    S11a["Step 11a: error mapping"]
    S11b["Step 11b: resolution facade"]
    S12["Step 12: streaming metadata merger"]
    S13["Step 13: collection over positions"]
    S14["Step 14: merged serve surface"]
    S15["Step 15: composition root"]
    S16a["Step 16a: merge memory cap"]
    S16b["Step 16b: S05 rate-limit wiring"]
    S17["Step 17: winner-path artifact and metadata handler"]
    S18["Step 18: sidecar handler by parent shape"]
    S19a["Step 19a: winner-path observability"]
    S19b["Step 19b: merge-path observability"]
    S20["Step 20: end-to-end harness and e2e catalog"]

    S1 --> S2
    S2 --> S3
    S3 --> S4
    S1 --> S8
    S8 --> S9
    S4 --> S10a
    S5 --> S10a
    S6 --> S10a
    S10a --> S10c
    S7 --> S10c
    S10b --> S10c
    S10c --> S11a
    S11a --> S11b
    S10c --> S13
    S12 --> S13
    S11a --> S13
    S13 --> S14
    S16a --> S14
    S19a --> S14
    S9 --> S15
    S11b --> S15
    S15 --> S17
    S18 --> S16b
    S14 --> S18
    S17 --> S18
    S17 --> S19a
    S19a --> S19b
    S18 --> S19b
    S17 --> S20
    S18 --> S20

Steps 1, 5, 6, 7, 10b, 12, and 16a declare Depends on: — and open at plan merge. Steps 12 and 16a stay parallel to the whole resolution track. Step 13 is held by the collection-mode plan's remaining steps (2, 4, 5, 6); Step 16b lands after Step 18 and is held by the S05 rate-limiting primitive amendment, gating nothing. #688 lands with or before Step 11a.

Status

Step Description MR Status Done Assignee Dependencies
1 maven_virtual_repositories schema — parent table, 64 partitions, unique index, S20-A FK collateral !2697 (merged) workflowcomplete @mkhalifa3 —
2 maven_virtual_repository_upstreams schema — junction table, position column, deferrable constraint, reverse index !2719 (merged) workflowcomplete @jdrpereira Step 1
3 maven_virtual_upstream_rules schema — per-association rule table with range and length CHECKs workflowin dev @sylviashen Step 2
4 Datastore reads for the upstream list and rules — batched reads, contract sidecar Step 3
5 Hosted upstream lookup — batched authoritative read over hosted Maven tables, Phase 1 adapter !2702 (merged) workflowcomplete @mkhalifa3 —
6 Remote upstream lookup and prober — batched cache reads, Phase 1 adapter, bare-HEAD prober !2703 (merged) workflowcomplete @mkhalifa3 —
7 Maven coordinate and rule adapter — target fields, coordinate from parsed path, stored-rule adapter !2690 (merged) workflowcomplete @mkhalifa3 —
8 Resolver virtual tier — Virtual field on Resolution, VirtualRepositoryFinder, wire_maven arity change !2737 workflowin review @mkhalifa3 Step 1
9 Virtual dispatch slot — kind=1 arm of serveDownload, FIPS .md5 short-circuit, write-path 405 pin Step 8
10a Memo adapters and the health reader — per-request adapters for fixed statement count Steps 4, 5, 6
10b The wrong-format data-error mark — Maven-side gate consuming S32's Step 10a declaration —; external: S32 Step 10a
10c Upstream assembly and the fail-closed gates — ordered position list, kind/format checks, soft-delete drop Steps 10a, 7, 10b
11a Error mapping — resolution error surface to Maven envelope, derived enumeration, coalescing sentinels Step 10c
11b Resolution facade — assemble, resolve, hand back outcome and position list Step 11a
12 Streaming metadata merger — pure merge, streaming token pass, deterministic output, escaping !2691 (merged) workflowcomplete @mkhalifa3 —
13 Collection over positions — contributor reads, PositionOutcome-to-token map, merge-path aggregate arms Steps 10c, 11a, 12; external: collection-mode plan steps 2, 4, 5, 6
14 Merged serve surface — validator, conditional path, exact framing, merge counters, cap slot Steps 13, 16a, 19a
15 Composition root — process-lifetime wiring, production audit sink, flight-registry seam Steps 9, 11b
16a Merge memory cap — byte-weighted in-flight merge budget under GOMEMLIMIT !2692 (merged) workflowcomplete @mkhalifa3 —
16b S05 rate-limit wiring — virtual read path through shared rate-limiting primitive, 429 shape Step 18; external: S05 amendment
17 Winner-path artifact and metadata handler — resolve, serve by winner kind, cache fallback, ADR-012 usage events Steps 11b, 15
18 Sidecar handler by parent shape — checksum sidecar and package-level metadata arms mounted, rate-limit registration Steps 14, 17
19a Winner-path observability — four metric families, maven_virtual_resolve wide event, label budgets Step 17
19b Merge-path observability — merge outcome values in wide event, skipped_soft_deleted records, catalog rows Steps 19a, 18
20 End-to-end harness and e2e catalog — hermetic harness, coalescing test, e2e scenario catalog Steps 17, 18

Cross-workstream dependencies (critical path)

  • S13 (virtual/remote foundation) — the resolution algorithm, health monitoring, and collection mode. The collection-mode plan's remaining steps (2, 4, 5, 6 of 2026-09-04-virtual-collection-mode.md) gate Step 13. Steps 1 and 3 of that plan have merged.
  • S14 (Maven remote, #286 (closed)) — fully merged. RemoteOperations, the RemoteCacheStore seams, BuildRemoteRequestBuilder, the bare-HEAD health probe, sidecar synthesis, and the maven_remote_proxy wide-event emitter are consumable as-is.
  • S10 (Maven hosted) — merged. S30 reuses internal/format/maven/'s path parser, coordinate grammar, Content-Type mapping, digest support, error envelope, and resolver/dispatcher.
  • S32 (container virtual) — S32's Step 10a declares the wrong-format exclusion reason (constructor, Reason classification, loadAndFilter arm, audit const) and is a merge-order prerequisite for this plan's Step 10b, which consumes it without redeclaring.
  • S05 (distributed state) — the rate-limiting primitive (ADR-004's Redis-counter, GCRA-based middleware) is a merge-order prerequisite for Step 16b. S30 consumes the primitive and mints no mechanism of its own.
  • #688 — sets Retry-After from remote.RetryAfterSeconds on the resolution path. Lands with or before Step 11a.
  • #314 — S17 Phase 6's Maven virtual management steps (junction store, association routes, delete guard) are gated on this plan's schema (Step 2). Same-release duty: the delete guard is never split from the association route.
  • ADR-007 handbook amendment — the ON DELETE actions and range/length CHECKs ADR-007 leaves unstated are a merge-order prerequisite for Steps 1-3. Filed against the handbook repository, tracked on #943.
  • npm's open Step 14 shares cmd/artifact-registry/remote_audit_sink.go with this plan's Step 15.
  • npm's open Step 18 and S32's Step 17 share the resolution_outcome closed-set union with this plan's Step 19a.

Related

Edited by Moaz Khalifa