S32: Container virtual implementation

Implement container virtual repositories per the S32 spec (#239). Depends on the container remote implementation (#288).

Spec and plan

Scope

The whole of S32. Container virtual (kind=1) vertical slice: the three container_virtual_* tables, the kind=1 arm of the OCI dispatcher, the loading slice that turns an upstream list into S13's resolution inputs, the container Probe and the token-cache read behind it, the shared per-repository single-flight the composition requires, the manifest and blob read handlers with their write-verb 405s and the two list-route 404s, the oci_virtual_* observability family, and the end-to-end harness.

Nothing here is new machinery for its own sake — everything it composes already exists. S12's hosted reads, S16's remote proxy, and S13's resolver in internal/virtual are all on main. What does not exist is anything that composes them for container, and the composition is the deliverable: a client pointing docker pull at one URL and receiving both the organization's own images and cached public ones, without knowing which repository answered.

Delivered as 19 steps, one MR each. Execution follows the dependency DAG below — not numeric order; each step's Depends on: names the predecessors whose MRs must merge before that step's MR opens. Six steps are roots (1, 5, 6, 7, 8, 13), so six authors can start the day the plan merges.

Shared machinery is numbered before its consumers. Steps 4 to 14 own every artifact more than one route needs. Step 14 wires both dispatch slots to an interim 501 before either handler exists, which is what makes Steps 15 and 16 siblings rather than a chain behind whichever route mounts first. This is shaped to avoid the failure S15 measured and S31 recorded: route-shaped steps opened in parallel, each rebuilding the same unowned foundation.

Three steps land code in internal/virtual, following the precedent S31 set: Step 9's probe-deadline split, Step 10a's two data-error causes, and Step 11's resolution deadline. Only the third needs an S13 spec amendment, and that amendment gates Step 11's MR. All three edit files S31 also edits, so each carries a cross-plan merge-order note.

Reachability: no step here delivers an operator-reachable capability until S17 Phase 6's Steps 38 and 42 land, and those are themselves gated on this plan's schema. Until then an association row arrives only by direct SQL, and this plan seeds its own — Step 14's seed_container_virtual recipe and Step 18's harness write into development and test databases that no purger reaps. The slice can merge complete and still be unusable rather than broken. This is a scheduling fact for the S32 and S17 DRIs, not a blocker on any step.

Out of scope: S13 owns the format-agnostic resolver, single-flight, freshness, and the SSRF-hardened upstream client. S16 owns container remote (kind=2) and its proxy handlers. S12 owns hosted container reads. S17 Phase 6 owns virtual-repo CRUD and the association surface. S20-A owns the repository-delete guard; S29 cache eviction; S08/S09 real auth.

Dependency DAG

graph TD
    ST1["✅ Step 1: container_virtual_repositories schema"]
    ST2["✅ Step 2: container_virtual_repository_upstreams schema"]
    ST3["👀 Step 3: container_virtual_upstream_rules schema"]
    ST4["🚫 Step 4: datastore reads for the upstream list and rules"]
    ST5["✅ Step 5: hosted upstream lookup"]
    ST6["✅ Step 6: remote upstream lookup"]
    ST7["✅ Step 7: container coordinate and rule adapter"]
    ST8["✅ Step 8: flight registry and per-call request state"]
    ST9["0️⃣ Step 9: container probe and the probe-deadline split"]
    ST10A["Step 10a: data-error positions in the foundation"]
    ST10B["Step 10b: upstream assembly and memoized lookups"]
    ST11["Step 11: resolution facade, error mapping, resolution deadline"]
    ST12["Step 12: virtual serve surface"]
    ST13["✅ Step 13: kind-dispatch seam, write 405s, list 404s"]
    ST14["Step 14: composition root"]
    ST15["Step 15: virtual manifest handler"]
    ST16["Step 16: virtual blob handler"]
    ST17["Step 17: observability"]
    ST18["Step 18: end-to-end harness and e2e catalog"]

    ST1 --> ST2
    ST2 --> ST3
    ST3 --> ST4
    ST8 --> ST9
    ST4 --> ST10B
    ST5 --> ST10B
    ST6 --> ST10B
    ST7 --> ST10B
    ST8 --> ST10B
    ST9 --> ST10A
    ST9 --> ST10B
    ST10A --> ST10B
    ST10B --> ST11
    ST11 --> ST12
    ST11 --> ST14
    ST12 --> ST14
    ST13 --> ST14
    ST14 --> ST15
    ST14 --> ST16
    ST15 --> ST17
    ST16 --> ST17
    ST15 --> ST18
    ST16 --> ST18

Status

The three schema MRs (Steps 1-3) are serialized, not parallel: internal/datastore/migrations/migrations.go sets goose.WithAllowOutofOrder(false) and a migration timestamp is fixed when the file is authored, so each takes the next free timestamp only after the previous one has merged. Both sibling slices serialized theirs for the same reason. Steps 2 and 3 also carry a composite foreign key into their predecessor's table, so the ordering is a data dependency as well as a timestamp one.

# Description MR Status Done Dependencies
1 Schema — container_virtual_repositories: the parent at 64 hash partitions, its composite primary key, its two foreign keys with their opposite delete actions, and its unique index !2159 (merged) workflowcomplete —
2 Schema — container_virtual_repository_upstreams: the join table, its two unique constraints — one of them the repository's first DEFERRABLE INITIALLY DEFERRED — its reverse index, its position >= 1 check, and the delete actions on its three foreign keys !2277 (merged) workflowcomplete Step 1; timestamp order after 1
3 Schema — container_virtual_upstream_rules: the three range and length CHECKs, the delete actions on its two foreign keys, and the index the batched rules read uses !2682 (merged) workflowin review Step 2; timestamp order after 2
4 Datastore reads for the upstream list and rules: the ordered upstream list joined to repositories and left-joined to container_remote_repositories, and the batched rules read across every association in that list. Both fixed-count, both bounded, both projecting no credential column Not started Step 3
5 Hosted upstream lookup: the batched authoritative reads for a hosted position — tag, manifest-by-digest, blob-by-digest, each across every hosted position in one statement — and the remote.Upstream that answers Phase 1 from them. Diffs against S31's merged hosted batched lookup for dropped subtests !2135 (merged) workflowcomplete —
6 Remote upstream lookup: the batched cache reads across the container remote cache tables, and the remote.Upstream answering Phase 1 from them with S13's freshness model — remote.LookupResult.ETag filled from the row's upstream_etag included, without which ProbeFresh is unreachable for every container position !2136 (merged) workflowcomplete —
7 Container coordinate and rule adapter: the defined target-field set, the per-request addressed values, and the container_virtual_upstream_rules row mapping onto []virtual.Rule. Pure mapping over internal/virtual's rule vocabulary !2118 (merged) workflowcomplete —
8 Flight registry and per-call request state: move all four of the container request builder's per-request values — accept, the url/auth_status/auth_url snapshot, imageName, and the row-rooted *UpstreamURLBuilder — onto a per-call context channel, and add the flight registry handing out one shared bounded *remote.SingleFlight per remote repository. Closes #596 and #752 !2685 (merged) workflowcomplete —; unblocks three S16 acceptance criteria gated on #752
9 Container probe and the probe-deadline split: a HEAD through remote.Prober carrying the pull-scoped token from S16's cache, the caller-supplied If-None-Match honored on the wire, the one-shot re-exchange on a 401, and the two budgets in sequence on independent deadlines. Plus the internal/virtual change: probeOne stops applying resolution.probe_timeout around the whole Probe and the Prober applies it at its HEAD leg Not started Step 8; no S13 amendment owed — S13's text already fixes the budget to the HEAD leg, so what moves is where the rule is applied in code. Cross-plan merge order against S31's Step 11
10a Data-error positions in the foundation: a constructor for a position that is data-error by construction, the two remote.UpstreamExclusionReason values, their classification, and the loadAndFilter arms that set st.dataError, log, and audit. No container code. Also collapses the four data-error arms to one Error line per resolution Not started Step 9 — file serialization on internal/virtual/resolve.go, not a data dependency. Cross-plan merge order against S31's Step 11
10b Upstream assembly and memoized lookups: turn a virtual repository and a target into []virtual.UpstreamSpec — drop the soft-deleted associations Step 4 flagged, prefetch the batched Lookups behind a memoized remote.Upstream, memoize health out of the list query's column, acquire each remote position's flight lazily, build each position's remote.CacheStore from the row already read, and reject a wrong-format or unbound-remote position through Step 10a's constructor rather than by dropping it Not started Steps 4, 5, 6, 7, 8, 9, 10a. Step 9 is a direct edge: it supplies the Prober each remote position carries
11 Resolution facade, error mapping, and the resolution deadline: one call answering a resolved read or a mapped error, the mapping from S13's resolution taxonomy onto the OCI envelope and the Retry-After classes, and the internal/virtual change the aggregate deadline needs — Resolver reads resolve_timeout off remote.ResolutionConfig and scopes a context.WithTimeoutCause child to Phases 1 and 2, handing every serve arm the caller's context Not started Step 10b; external gate: the S13 spec amendment defining virtual_repositories.resolution.resolve_timeout must merge before this MR opens. Cross-plan merge order against S31's Step 12
12 Virtual serve surface: the shared manifest and blob response construction over both winner kinds — build the headers from whichever row won, negotiate content on the resolved manifest's stored media_type, and hand a blob to S06's serve path so ADR-005's delivery mode decides 200 against 307. Branches on winner kind to choose the header source, not the delivery mechanism Not started Step 11
13 Kind-dispatch seam, write 405s, list 404s: replace the dispatcher's single virtual 501 branch with a route table — two read slots left unset, 405 UNSUPPORTED with the right Allow on every write verb, 404 NAME_UNKNOWN on tags/list and referrers, and the Cache-Control the replaced branch carried !2138 (merged) workflowcomplete —; the ADR-009 amendment annotating tags/list and referrers as unavailable on a virtual repository is filed before this MR opens
14 Composition root: wire_oci_virtual.go with both slots wired to the interim 501, the process-wide virtual.Resolver and its memoizing health reader, the container flight registry, the production exclusion audit sink, and the per-route builder functions Steps 15 and 16 fill. Plus rewiring the standalone container remote read path onto the same registry Not started Steps 11, 12, 13
15 Virtual manifest handler: GET and HEAD on the manifest route for a kind=1 repository, swapping the interim 501 for a real resolution, plus the write-deadline arming this route needs and no virtual route has Not started Step 14. Sibling of 16
16 Virtual blob handler: GET and HEAD on the blob route for a kind=1 repository, plus its own write-deadline arming — without it no cold layer over ten seconds ever caches, because a virtual route otherwise runs under server.timeouts.write Not started Step 14; external: #433's tag-addressed half — that subtest ships written and skipped. Sibling of 15
17 Observability: the three oci_virtual_* metric families, the oci_virtual_resolve wide event with its positions[] array, and the two extensions to S16's instruments. Reaches positions[] by wrapping five seams this plan supplies — Lookup, Prober, Fetcher, Cache, and the memoizing remote.HealthStatusReader Not started Steps 15, 16; no external gate — S16 Step 17 has merged, so the probe extensions have something to extend
18 End-to-end harness and e2e catalog: a hermetic in-package harness driving a real pull journey — index, child manifest, and layers — across a mixed hosted-and-remote upstream list, plus the catalog sections. No live upstream, no separate CI job Not started Steps 15, 16

Cross-workstream dependencies (critical path)

  • S13 resolution deadline amendment — the one external gate that delays an MR. An S13 spec amendment defining virtual_repositories.resolution.resolve_timeout in the shared block gates Step 11. It is the one amendment that opens before the step it serves, because the key lands in the virtual_repositories block S30, S31, and S32 share rather than on the container slice. S31 books the same amendment for the same key; whichever plan files it first, both consume it. The S13 DRI ruled the deadline into the foundation: no caller can express a budget that ends at winner selection, because a single-flight follower's wait happens inside Resolve, and context.WithCancel plus time.AfterFunc cannot tell a budget expiry from a client hangup. The value is 90s, covering container's 60s worst case — two waves at probe_concurrency=10 against ADR-004's cap, at 10s + 5s + 10s + 5s per cold-token probe carrying a 401 — with real headroom, and it costs S31 nothing. Owner: the S13 DRI.
  • S32 spec amendment reversing the ## Configuration ruling. The spec says three times that S32 introduces no configuration and that a per-read budget is deliberately not added; Step 11 adds one, because nothing on the server side holds the client deadline the spec names as the bound. The same amendment adds the Error Cases row and Retry-After class the new 503 needs — 503 UNAVAILABLE with a handler-supplied value, on the digest-mismatch model — and the sixteenth resolution_outcome value, resolve_deadline_exceeded. That last one cannot ship folded under unavailable, which would merge a self-inflicted timeout with a real upstream outage, and it cannot ship unnamed, because internal/metrics/cardinality.go fails its audit on a closed-set value outside its set. Rides with the S13 amendment; gates no step of its own.
  • S32 spec amendment for the Metrics table, gating no step: operation becomes oci_virtual_operation on all three families, and outcome becomes resolution_outcome on resolutions_total, the only family carrying an outcome label. A scraped label cannot be renamed after it ships, so what the amendment prevents is a spec describing labels no dashboard will find.
  • One name for the not-found branch across the two virtual slices, settled before either observability step opens. internal/metrics/cardinality.go keys its closed sets by label name across the whole service, so S31 and S32 share one resolution_outcome entry whatever either plan intends — and their vocabularies differ: S31 calls the branch no_winner, S32 calls it not_found_on_upstreams. Shipping both puts one branch under two names on a label operators filter across formats. Both plans are in flight, so this is the window. Gates no step; what it decides is whether the union has 16 members or 17. Owner: the S13 DRI.
  • Retry-After ratification for the data-error veto (#875). S32's veto criterion must not be asserted before #875 lands: S13's rule as it stands assigns that row health_check.scheduled_interval, so a test written to the criterion today fails S13's own. Step 11 asserts the veto row's status and UpstreamUnavailable mapping only. That row has a second gate #875 does not close — the resolver returns one ErrUpstreamUnavailable for both the veto 503 and a plain outage 503, and telling them apart needs a per-position record, which this plan reaches in Step 17. The header assertion rides a follow-up behind the later of the two.
  • S13 fetch-seam cross-check (#433). The by-digest half has landed — FetchOptions.ExpectedDigest reaches the fill's commit — so Step 16 asserts that criterion. The tag-addressed half needs a verdict a slice can act on and is not observable yet, so Step 16's tag-mismatch subtest is written and skipped with the work item named. S16 gates its own equivalent criteria the same way.
  • S13 amendment !2013 (merged) governs two things this plan rests on. A new RequestBuilder clause requires reading the remote repository's stored credentials and url inside BuildRequest on every call, never capturing them at construction — that is Step 8's premise stated as a requirement, and Step 8's MR description states its reading against the clause. A new single-flight clause fixes that live teeing is available only until the leader forwards its first byte, so Step 15's coalescing test and Step 16's different-blobs criterion are written against the committed-document path.
  • S16 remote is merged bar one part, and neither open branch is a prerequisite. Everything Steps 6, 8, 9, 12, and 16 consume is on main. Two S16 branches are files this plan edits, so both belong on the git diff --name-only main...HEAD list Steps 8 and 13 run: !1904 (merged), the last Step 15 part carrying S16's own composition root, and !1875 (merged), the fourth manifest-proxy part. The manifest proxy splits four ways, not three — the S16 plan carries Step 14 as a single Status row and its sizing paragraph gives figures for three parts, so a reader taking that breakdown as the split leaves !1875 (merged) off the overlap list.
  • Two work items this plan closes rather than waits on. #752, the container single-flight composition, and #596, the request-builder per-call channel, are both Step 8's. #596's own text names two values and there are four, so Step 8 corrects the work item and the type's doc comment alongside the code. Three of S16's acceptance criteria are gated on #752, and #320 (closed), which held the third back, has closed — so Step 8's MR unblocks all three. The S16 DRI should know that.
  • S17 Phase 6's Steps 38 and 42 are not prerequisites, and cannot be: their own dependency graph gates them on this plan's schema. Merging these handlers ahead of them is safe against the management API and leaves the feature unusable rather than broken. Step 42 is also the container arm of the repository-delete guard, and the spec states over the row rather than the schema that no association row is written until that guard runs, because container_virtual_repository_upstreams's NO ACTION foreign key stalls S20-A's purger from the moment the first association exists.
  • The production remote.UpstreamExclusionAuditSink may arrive from S31's Step 14 first. Step 14's shared-seam entry records the method name and signature both plans write, so the two declarations collapse rather than conflict; whichever lands second wires it.
  • #867, the fourth data-error shape Steps 10a and 10b do not close. A repositories row whose format is docker or oci with no container_repositories child passes both the kind gate and the format gate, is consulted as a hosted upstream, and misses by construction. A hosted miss establishes an absence, so the answer is a permanent 404 — the fail-open those two steps exist to turn into a 500, through a shape neither gate can see. Both candidate fixes bind every format, so it is the S13 DRI's to settle. Gates no step; Step 10b's acceptance carries the pointer so the three-gate list is not read as complete.
  • A work item against internal/virtual for bounding data-error log volume. Step 10a removes the position multiplier but not the per-layer one: a container pull is L + 2 resolutions, so one persistent bad row still writes about 52 lines per client per pull, for as long as the row exists. A sampled line or a throttle keyed on (repository, cause) belongs in the foundation — every virtual slice inherits the shape, and container is simply the first with a per-layer read pattern to make it visible. Owner: the S13 DRI.
  • #912, collapsing the two standalone write-deadline arming helpers. Steps 15 and 16 add two more arming sites on budgets of their own, so whichever of #912 and this plan moves second inherits four call sites rather than two. Step 14's wiring puts the standalone and virtual arms behind one flight registry, so a fill either starts is one the other can be waiting on. Gates nothing. Owner: the S16 DRI.
  • A work item against S16's UpstreamAuthenticator for coalescing challenge discovery. Discovery is not coalesced, so a burst of C clients across M auth_status=unknown upstreams costs C x M discovery HEADs and C x M auth_discovery_writes UPDATEs. S32 is the first slice to multiply the write half by the upstream count. Gates nothing: Step 9 names the existing counter as the signal and ships. Owner: the S16 DRI.
  • A per-position observation seam on internal/virtual, as a non-blocking cross-slice follow-up. This plan reaches positions[] by wrapping five seams; S31 wraps the same five and S30 would be the third. A foundation seam would let the three slices share one implementation and would reach the resolver's own cross-position verdict ordering, which no wrapper can see. If it lands after Step 17, Step 17's wrappers collapse into it as a simplification. Owner: the S13 DRI.
  • Two handbook amendment MRs, gating no step but filed before the step that lands each divergence opens: an ADR-007 amendment for the ## Data Model items, and an ADR-009 amendment annotating tags/list and referrers as unavailable on a virtual repository. The filings precede Step 1 and Step 13 respectively, because those MR descriptions carry the URLs and a description promising a URL that does not exist yet is how a divergence ships with no amendment on record. docs/adr/ cannot be edited from this repository.
  • The cold-pull probe amplification is unmonitored until Step 17. The spec accepts it for the MVP conditionally and nominates gitlab_artifact_registry_oci_virtual_probes_total as the signal that says whether it needs closing. The real figure is C x M x (L + 2) rather than the spec's per-resolution C x M: a cold 50-layer pull against three remote upstreams is 156 upstream HEADs. That counter is Step 17's, so between Steps 15 and 16 merging and Step 17 merging the acceptance holds but the evidence that would revisit it does not exist yet. Interim coverage is the S03-B HTTP server family alone.
  • S32's two Open Questions are open above this plan and change nothing in it. Whether an upstream may be less visible than the virtual repository listing it (for the S17 and S13 DRIs), and whether the association surface should stop a public remote outranking hosted content (#792, GA-blocking). Every candidate answer sits on an S17 write surface. This plan's read path adds no visibility gate for that reason, not by omission; if the first question is answered with a read-side control after all, it lands as a gate in Step 10b.
  • Authorization is a stub on main. Step 13's ordering criterion is asserted against an injected denying decision, not the stub.

Follow-Ups

Work this initiative surfaced that no S32 step MR carries. Each entry says what holds until it lands, so the list reads correctly whether or not it has been picked up.

  • Two stale lock and rollback claims the virtual tables invalidated — #1027. A shipped npm virtual Down tells an operator the rollback entrypoint leaves no session to issue a SET in, which goose disproves and docs/dev/database-migrations.md now contradicts, and that Down's parent DROP takes ACCESS EXCLUSIVE on 131 relations unbounded. Separately, NamespaceStore's measured lock-budget figures move by about 65 relations per virtual table and are stale by one table per landed step. Until #1027 lands, read the migration's line 642 as wrong and the namespaces.go figures as lower bounds. Correcting the migration needs an Allow-Migration-Edit trailer.

Related

Edited by Radamanthus Batnag