Implement container virtual repositories per the S32 spec (#239). Depends on the container remote implementation (#288). ## Spec and plan - Spec: [`docs/specs/S32-container-virtual.md`](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/S32-container-virtual.md) - Plan: [`docs/plans/2026-08-27-s32-container-virtual.md`](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/plans/2026-08-27-s32-container-virtual.md) ## Scope The whole of S32. Container virtual (`kind=1`) vertical slice: the three `container_virtual_*` tables, the `kind=1` arm of the OCI dispatcher, the loading slice that turns an upstream list into S13's resolution inputs, the container Probe and the token-cache read behind it, the shared per-repository single-flight the composition requires, the manifest and blob read handlers with their write-verb `405`s and the two list-route `404`s, the `oci_virtual_*` observability family, and the end-to-end harness. Nothing here is new machinery for its own sake — everything it composes already exists. S12's hosted reads, S16's remote proxy, and S13's resolver in `internal/virtual` are all on `main`. What does not exist is anything that composes them for container, and the composition is the deliverable: a client pointing `docker pull` at one URL and receiving both the organization's own images and cached public ones, without knowing which repository answered. Delivered as **19 steps, one MR each**. Execution follows the dependency DAG below — **not numeric order**; each step's `Depends on:` names the predecessors whose MRs must merge before that step's MR opens. Six steps are roots (1, 5, 6, 7, 8, 13), so six authors can start the day the plan merges. **Shared machinery is numbered before its consumers.** Steps 4 to 14 own every artifact more than one route needs. Step 14 wires both dispatch slots to an interim `501` before either handler exists, which is what makes Steps 15 and 16 siblings rather than a chain behind whichever route mounts first. This is shaped to avoid the failure S15 measured and S31 recorded: route-shaped steps opened in parallel, each rebuilding the same unowned foundation. **Three steps land code in `internal/virtual`**, following the precedent S31 set: Step 9's probe-deadline split, Step 10a's two data-error causes, and Step 11's resolution deadline. Only the third needs an S13 spec amendment, and that amendment gates Step 11's MR. All three edit files S31 also edits, so each carries a cross-plan merge-order note. **Reachability:** no step here delivers an operator-reachable capability until S17 Phase 6's Steps 38 and 42 land, and those are themselves gated on this plan's schema. Until then an association row arrives only by direct SQL, and this plan seeds its own — Step 14's `seed_container_virtual` recipe and Step 18's harness write into development and test databases that no purger reaps. The slice can merge complete and still be unusable rather than broken. This is a scheduling fact for the S32 and S17 DRIs, not a blocker on any step. **Out of scope:** S13 owns the format-agnostic resolver, single-flight, freshness, and the SSRF-hardened upstream client. S16 owns container remote (`kind=2`) and its proxy handlers. S12 owns hosted container reads. S17 Phase 6 owns virtual-repo CRUD and the association surface. S20-A owns the repository-delete guard; S29 cache eviction; S08/S09 real auth. ## Dependency DAG ```mermaid graph TD ST1["Step 1: container_virtual_repositories schema"] ST2["Step 2: container_virtual_repository_upstreams schema"] ST3["Step 3: container_virtual_upstream_rules schema"] ST4["Step 4: datastore reads for the upstream list and rules"] ST5["Step 5: hosted upstream lookup"] ST6["Step 6: remote upstream lookup"] ST7["Step 7: container coordinate and rule adapter"] ST8["Step 8: flight registry and per-call request state"] ST9["Step 9: container probe and the probe-deadline split"] ST10A["Step 10a: data-error positions in the foundation"] ST10B["Step 10b: upstream assembly and memoized lookups"] ST11["Step 11: resolution facade, error mapping, resolution deadline"] ST12["Step 12: virtual serve surface"] ST13["Step 13: kind-dispatch seam, write 405s, list 404s"] ST14["Step 14: composition root"] ST15["Step 15: virtual manifest handler"] ST16["Step 16: virtual blob handler"] ST17["Step 17: observability"] ST18["Step 18: end-to-end harness and e2e catalog"] ST1 --> ST2 ST2 --> ST3 ST3 --> ST4 ST8 --> ST9 ST4 --> ST10B ST5 --> ST10B ST6 --> ST10B ST7 --> ST10B ST8 --> ST10B ST9 --> ST10A ST9 --> ST10B ST10A --> ST10B ST10B --> ST11 ST11 --> ST12 ST11 --> ST14 ST12 --> ST14 ST13 --> ST14 ST14 --> ST15 ST14 --> ST16 ST15 --> ST17 ST16 --> ST17 ST15 --> ST18 ST16 --> ST18 ``` ## Status The three schema MRs (Steps 1-3) are **serialized, not parallel**: `internal/datastore/migrations/migrations.go` sets `goose.WithAllowOutofOrder(false)` and a migration timestamp is fixed when the file is authored, so each takes the next free timestamp only _after the previous one has merged_. Both sibling slices serialized theirs for the same reason. Steps 2 and 3 also carry a composite foreign key into their predecessor's table, so the ordering is a data dependency as well as a timestamp one. <table> <tr> <th> \# </th> <th>Description</th> <th>MR</th> <th>Status</th> <th>Done</th> <th>Dependencies</th> </tr> <tr> <td>1</td> <td> Schema — `container_virtual_repositories`: the parent at 64 hash partitions, its composite primary key, its two foreign keys with their opposite delete actions, and its unique index </td> <td></td> <td>Not started</td> <td> * [ ] </td> <td>—</td> </tr> <tr> <td>2</td> <td> Schema — `container_virtual_repository_upstreams`: the join table, its two unique constraints — one of them the repository's first `DEFERRABLE INITIALLY DEFERRED` — its reverse index, its `position >= 1` check, and the delete actions on its **three** foreign keys </td> <td></td> <td>Not started</td> <td> * [ ] </td> <td>Step 1; timestamp order after 1</td> </tr> <tr> <td>3</td> <td> Schema — `container_virtual_upstream_rules`: the three range and length `CHECK`s, the delete actions on its **two** foreign keys, and the index the batched rules read uses </td> <td></td> <td>Not started</td> <td> * [ ] </td> <td>Step 2; timestamp order after 2</td> </tr> <tr> <td>4</td> <td> Datastore reads for the upstream list and rules: the ordered upstream list joined to `repositories` and left-joined to `container_remote_repositories`, and the batched rules read across every association in that list. Both fixed-count, both bounded, both projecting no credential column </td> <td></td> <td>Not started</td> <td> * [ ] </td> <td>Step 3</td> </tr> <tr> <td>5</td> <td> Hosted upstream lookup: the batched authoritative reads for a hosted position — tag, manifest-by-digest, blob-by-digest, each across every hosted position in one statement — and the `remote.Upstream` that answers Phase 1 from them. Diffs against S31's merged hosted batched lookup for dropped subtests </td> <td></td> <td>Not started</td> <td>—</td> </tr> <tr> <td>6</td> <td> Remote upstream lookup: the batched cache reads across the container remote cache tables, and the `remote.Upstream` answering Phase 1 from them with S13's freshness model — **`remote.LookupResult.ETag` filled from the row's `upstream_etag` included**, without which `ProbeFresh` is unreachable for every container position </td> <td></td> <td>Not started</td> <td>—</td> </tr> <tr> <td>7</td> <td> Container coordinate and rule adapter: the defined target-field set, the per-request addressed values, and the `container_virtual_upstream_rules` row mapping onto `[]virtual.Rule`. Pure mapping over `internal/virtual`'s rule vocabulary </td> <td></td> <td>Not started</td> <td>—</td> </tr> <tr> <td>8</td> <td> Flight registry and per-call request state: move **all four** of the container request builder's per-request values — `accept`, the `url`/`auth_status`/`auth_url` snapshot, `imageName`, and the row-rooted `*UpstreamURLBuilder` — onto a per-call context channel, and add the flight registry handing out one shared bounded `*remote.SingleFlight` per remote repository. Closes [#596](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/596) and [#752](https://gitlab.com/gitlab-org/ops/artifact-registry/-/issues/752) </td> <td></td> <td>Not started</td> <td> —; unblocks three S16 acceptance criteria gated on #752 </td> </tr> <tr> <td>9</td> <td> Container probe and the probe-deadline split: a `HEAD` through `remote.Prober` carrying the pull-scoped token from S16's cache, the caller-supplied `If-None-Match` honored on the wire, the one-shot re-exchange on a `401`, and the two budgets in sequence on independent deadlines. Plus the `internal/virtual` change: `probeOne` stops applying `resolution.probe_timeout` around the whole Probe and the Prober applies it at its `HEAD` leg </td> <td></td> <td>Not started</td> <td> Step 8; no S13 amendment owed — S13's text already fixes the budget to the `HEAD` leg, so what moves is where the rule is applied in code. Cross-plan merge order against S31's Step 11 </td> </tr> <tr> <td>10a</td> <td> Data-error positions in the foundation: a constructor for a position that is data-error by construction, the two `remote.UpstreamExclusionReason` values, their classification, and the `loadAndFilter` arms that set `st.dataError`, log, and audit. No container code. Also collapses the four data-error arms to one Error line per resolution </td> <td></td> <td>Not started</td> <td> Step 9 — file serialization on `internal/virtual/resolve.go`, not a data dependency. Cross-plan merge order against S31's Step 11 </td> </tr> <tr> <td>10b</td> <td> Upstream assembly and memoized lookups: turn a virtual repository and a target into `[]virtual.UpstreamSpec` — drop the soft-deleted associations Step 4 flagged, prefetch the batched Lookups behind a memoized `remote.Upstream`, memoize health out of the list query's column, acquire each remote position's flight lazily, build each position's `remote.CacheStore` from the row already read, and reject a wrong-format or unbound-remote position through Step 10a's constructor rather than by dropping it </td> <td></td> <td>Not started</td> <td> Steps 4, 5, 6, 7, 8, 9, 10a. Step 9 is a **direct** edge: it supplies the Prober each remote position carries </td> </tr> <tr> <td>11</td> <td> Resolution facade, error mapping, and the resolution deadline: one call answering a resolved read or a mapped error, the mapping from S13's resolution taxonomy onto the OCI envelope and the `Retry-After` classes, and the `internal/virtual` change the aggregate deadline needs — `Resolver` reads `resolve_timeout` off `remote.ResolutionConfig` and scopes a `context.WithTimeoutCause` child to Phases 1 and 2, handing every serve arm the caller's context </td> <td></td> <td>Not started</td> <td> Step 10b; **external gate:** the S13 spec amendment defining `virtual_repositories.resolution.resolve_timeout` must merge before this MR opens. Cross-plan merge order against S31's Step 12 </td> </tr> <tr> <td>12</td> <td> Virtual serve surface: the shared manifest and blob response construction over both winner kinds — build the headers from whichever row won, negotiate content on the resolved manifest's stored `media_type`, and hand a blob to S06's serve path so ADR-005's delivery mode decides `200` against `307`. Branches on winner kind to choose the header source, not the delivery mechanism </td> <td></td> <td>Not started</td> <td>Step 11</td> </tr> <tr> <td>13</td> <td> Kind-dispatch seam, write `405`s, list `404`s: replace the dispatcher's single virtual `501` branch with a route table — two read slots left unset, `405 UNSUPPORTED` with the right `Allow` on every write verb, `404 NAME_UNKNOWN` on `tags/list` and `referrers`, and the `Cache-Control` the replaced branch carried </td> <td></td> <td>Not started</td> <td> —; the ADR-009 amendment annotating `tags/list` and `referrers` as unavailable on a virtual repository is filed before this MR opens </td> </tr> <tr> <td>14</td> <td> Composition root: `wire_oci_virtual.go` with both slots wired to the interim `501`, the process-wide `virtual.Resolver` and its memoizing health reader, the container flight registry, the production exclusion audit sink, and the per-route builder functions Steps 15 and 16 fill. Plus rewiring the **standalone** container remote read path onto the same registry </td> <td></td> <td>Not started</td> <td>Steps 11, 12, 13</td> </tr> <tr> <td>15</td> <td> Virtual manifest handler: `GET` and `HEAD` on the manifest route for a `kind=1` repository, swapping the interim `501` for a real resolution, plus the write-deadline arming this route needs and no virtual route has </td> <td></td> <td>Not started</td> <td>Step 14. Sibling of 16</td> </tr> <tr> <td>16</td> <td> Virtual blob handler: `GET` and `HEAD` on the blob route for a `kind=1` repository, plus its own write-deadline arming — without it no cold layer over ten seconds ever caches, because a virtual route otherwise runs under `server.timeouts.write` </td> <td></td> <td>Not started</td> <td> Step 14; external: [#433](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/433)'s tag-addressed half — that subtest ships written and skipped. Sibling of 15 </td> </tr> <tr> <td>17</td> <td> Observability: the three `oci_virtual_*` metric families, the `oci_virtual_resolve` wide event with its `positions[]` array, and the two extensions to S16's instruments. Reaches `positions[]` by wrapping five seams this plan supplies — Lookup, Prober, Fetcher, Cache, and the memoizing `remote.HealthStatusReader` </td> <td></td> <td>Not started</td> <td> Steps 15, 16; no external gate — S16 Step 17 has merged, so the `probe` extensions have something to extend </td> </tr> <tr> <td>18</td> <td> End-to-end harness and e2e catalog: a hermetic in-package harness driving a real pull journey — index, child manifest, and layers — across a mixed hosted-and-remote upstream list, plus the catalog sections. No live upstream, no separate CI job </td> <td></td> <td>Not started</td> <td>Steps 15, 16</td> </tr> </table> ## Cross-workstream dependencies (critical path) - **S13 resolution deadline amendment — the one external gate that delays an MR.** An S13 spec amendment defining `virtual_repositories.resolution.resolve_timeout` in the shared block gates Step 11. It is the one amendment that opens **before** the step it serves, because the key lands in the `virtual_repositories` block S30, S31, and S32 share rather than on the container slice. S31 books the same amendment for the same key; whichever plan files it first, both consume it. The S13 DRI ruled the deadline into the foundation: no caller can express a budget that ends at winner selection, because a single-flight follower's wait happens inside `Resolve`, and `context.WithCancel` plus `time.AfterFunc` cannot tell a budget expiry from a client hangup. The value is `90s`, covering container's 60s worst case — two waves at `probe_concurrency=10` against ADR-004's cap, at 10s + 5s + 10s + 5s per cold-token probe carrying a `401` — with real headroom, and it costs S31 nothing. Owner: the S13 DRI. - **S32 spec amendment reversing the `## Configuration` ruling.** The spec says three times that S32 introduces no configuration and that a per-read budget is deliberately not added; Step 11 adds one, because nothing on the server side holds the client deadline the spec names as the bound. The same amendment adds the Error Cases row and `Retry-After` class the new `503` needs — `503 UNAVAILABLE` with a handler-supplied value, on the digest-mismatch model — and the sixteenth `resolution_outcome` value, `resolve_deadline_exceeded`. That last one cannot ship folded under `unavailable`, which would merge a self-inflicted timeout with a real upstream outage, and it cannot ship unnamed, because `internal/metrics/cardinality.go` fails its audit on a closed-set value outside its set. Rides with the S13 amendment; gates no step of its own. - **S32 spec amendment for the Metrics table**, gating no step: `operation` becomes `oci_virtual_operation` on all three families, and `outcome` becomes `resolution_outcome` on `resolutions_total`, the only family carrying an outcome label. A scraped label cannot be renamed after it ships, so what the amendment prevents is a spec describing labels no dashboard will find. - **One name for the not-found branch across the two virtual slices**, settled before either observability step opens. `internal/metrics/cardinality.go` keys its closed sets by label name across the whole service, so S31 and S32 share one `resolution_outcome` entry whatever either plan intends — and their vocabularies differ: S31 calls the branch `no_winner`, S32 calls it `not_found_on_upstreams`. Shipping both puts one branch under two names on a label operators filter across formats. Both plans are in flight, so this is the window. Gates no step; what it decides is whether the union has 16 members or 17. Owner: the S13 DRI. - **`Retry-After` ratification for the data-error veto** ([#875](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/875)). S32's veto criterion must not be asserted before #875 lands: S13's rule as it stands assigns that row `health_check.scheduled_interval`, so a test written to the criterion today fails S13's own. Step 11 asserts the veto row's status and `UpstreamUnavailable` mapping only. That row has a second gate #875 does not close — the resolver returns one `ErrUpstreamUnavailable` for both the veto `503` and a plain outage `503`, and telling them apart needs a per-position record, which this plan reaches in Step 17. The header assertion rides a follow-up behind the later of the two. - **S13 fetch-seam cross-check** ([#433](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/433)). The by-digest half has landed — `FetchOptions.ExpectedDigest` reaches the fill's commit — so Step 16 asserts that criterion. The tag-addressed half needs a verdict a slice can act on and is not observable yet, so Step 16's tag-mismatch subtest is written and skipped with the work item named. S16 gates its own equivalent criteria the same way. - **S13 amendment [!2013 (merged)](https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/2013) governs two things this plan rests on.** A new `RequestBuilder` clause requires reading the remote repository's stored credentials and `url` inside `BuildRequest` on every call, never capturing them at construction — that is Step 8's premise stated as a requirement, and Step 8's MR description states its reading against the clause. A new single-flight clause fixes that live teeing is available only until the leader forwards its first byte, so Step 15's coalescing test and Step 16's different-blobs criterion are written against the committed-document path. - **S16 remote is merged bar one part, and neither open branch is a prerequisite.** Everything Steps 6, 8, 9, 12, and 16 consume is on `main`. Two S16 branches are files this plan edits, so both belong on the `git diff --name-only main...HEAD` list Steps 8 and 13 run: [!1904](https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1904), the last Step 15 part carrying S16's own composition root, and [!1875](https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1875), the fourth manifest-proxy part. **The manifest proxy splits four ways, not three** — the S16 plan carries Step 14 as a single Status row and its sizing paragraph gives figures for three parts, so a reader taking that breakdown as the split leaves !1875 off the overlap list. - **Two work items this plan closes rather than waits on.** [#752](https://gitlab.com/gitlab-org/ops/artifact-registry/-/issues/752), the container single-flight composition, and [#596](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/596), the request-builder per-call channel, are both Step 8's. #596's own text names two values and there are four, so Step 8 corrects the work item and the type's doc comment alongside the code. Three of S16's acceptance criteria are gated on #752, and [#320](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/320), which held the third back, has closed — so Step 8's MR unblocks all three. The S16 DRI should know that. - **S17 Phase 6's Steps 38 and 42 are not prerequisites**, and cannot be: their own dependency graph gates them on this plan's schema. Merging these handlers ahead of them is safe against the management API and leaves the feature unusable rather than broken. Step 42 is also the container arm of the repository-delete guard, and the spec states over the row rather than the schema that no association row is written until that guard runs, because `container_virtual_repository_upstreams`'s `NO ACTION` foreign key stalls S20-A's purger from the moment the first association exists. - **The production `remote.UpstreamExclusionAuditSink`** may arrive from S31's Step 14 first. Step 14's shared-seam entry records the method name and signature both plans write, so the two declarations collapse rather than conflict; whichever lands second wires it. - **[#867](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/867), the fourth data-error shape Steps 10a and 10b do not close.** A `repositories` row whose `format` is docker or oci with no `container_repositories` child passes both the `kind` gate and the `format` gate, is consulted as a hosted upstream, and misses by construction. A hosted miss establishes an absence, so the answer is a permanent `404` — the fail-open those two steps exist to turn into a `500`, through a shape neither gate can see. Both candidate fixes bind every format, so it is the S13 DRI's to settle. Gates no step; Step 10b's acceptance carries the pointer so the three-gate list is not read as complete. - **A work item against `internal/virtual` for bounding data-error log volume.** Step 10a removes the position multiplier but not the per-layer one: a container pull is L + 2 resolutions, so one persistent bad row still writes about 52 lines per client per pull, for as long as the row exists. A sampled line or a throttle keyed on `(repository, cause)` belongs in the foundation — every virtual slice inherits the shape, and container is simply the first with a per-layer read pattern to make it visible. Owner: the S13 DRI. - **[#912](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/912), collapsing the two standalone write-deadline arming helpers.** Steps 15 and 16 add two more arming sites on budgets of their own, so whichever of #912 and this plan moves second inherits four call sites rather than two. Step 14's wiring puts the standalone and virtual arms behind one flight registry, so a fill either starts is one the other can be waiting on. Gates nothing. Owner: the S16 DRI. - **A work item against S16's `UpstreamAuthenticator` for coalescing challenge discovery.** Discovery is not coalesced, so a burst of C clients across M `auth_status=unknown` upstreams costs C x M discovery `HEAD`s and C x M `auth_discovery_writes` UPDATEs. S32 is the first slice to multiply the write half by the upstream count. Gates nothing: Step 9 names the existing counter as the signal and ships. Owner: the S16 DRI. - **A per-position observation seam on `internal/virtual`**, as a non-blocking cross-slice follow-up. This plan reaches `positions[]` by wrapping five seams; S31 wraps the same five and S30 would be the third. A foundation seam would let the three slices share one implementation and would reach the resolver's own cross-position verdict ordering, which no wrapper can see. If it lands after Step 17, Step 17's wrappers collapse into it as a simplification. Owner: the S13 DRI. - **Two handbook amendment MRs**, gating no step but **filed before the step that lands each divergence opens**: an ADR-007 amendment for the `## Data Model` items, and an ADR-009 amendment annotating `tags/list` and `referrers` as unavailable on a virtual repository. The filings precede Step 1 and Step 13 respectively, because those MR descriptions carry the URLs and a description promising a URL that does not exist yet is how a divergence ships with no amendment on record. `docs/adr/` cannot be edited from this repository. - **The cold-pull probe amplification is unmonitored until Step 17.** The spec accepts it for the MVP *conditionally* and nominates `gitlab_artifact_registry_oci_virtual_probes_total` as the signal that says whether it needs closing. The real figure is C x M x (L + 2) rather than the spec's per-resolution C x M: a cold 50-layer pull against three remote upstreams is 156 upstream `HEAD`s. That counter is Step 17's, so between Steps 15 and 16 merging and Step 17 merging the acceptance holds but the evidence that would revisit it does not exist yet. Interim coverage is the S03-B HTTP server family alone. - **S32's two Open Questions are open above this plan and change nothing in it.** Whether an upstream may be less visible than the virtual repository listing it (for the S17 and S13 DRIs), and whether the association surface should stop a public remote outranking hosted content ([#792](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/792), **GA-blocking**). Every candidate answer sits on an S17 write surface. This plan's read path adds no visibility gate for that reason, not by omission; if the first question is answered with a read-side control after all, it lands as a gate in Step 10b. - **Authorization is a stub** on `main`. Step 13's ordering criterion is asserted against an injected denying decision, not the stub. ## Related - Spec: #239 - Foundation: #280 - Container remote (`kind=2`): #288 - npm virtual sibling: #290