S16: Container remote implementation

Implement container remote repositories per the S16 spec (#283 (closed)). Depends on the S13 foundation implementation (#280 (closed)).

Spec and plan

Scope

Container remote (kind=2) vertical slice: the six container_remote_* cache tables, ContainerRemoteConfig, the WWW-Authenticate challenge parser and token-exchange handshake, the token cache, the read-path kind-dispatch seam, the cache-hit serve path, the manifest and blob proxy paths, the two live-proxied lists (tag listing and referrers), download signals, the health-probe contribution, and the oci_remote_* observability surface.

Delivered as 18 implementation steps across 22 MRs — Steps 1 and 2 each ship three sequential MRs, one table per MR; every other step is one MR. Two tracks, split at the first upstream byte: Track A (Steps 1-11) consumes merged S13 surface and can open today; Track B (Steps 12-18) is downstream of a real end-to-end Fetch and reaches its one external prerequisite (S13 Step 11) through Step 12. Execution follows the dependency DAG below — not numeric order; each step's blockers are its direct predecessors. Step numbers are stable references for MR titles and review.

All 110 of the spec's acceptance criteria have a named owning step; the plan's Acceptance-criterion ownership table maps them, and marks the ones that ship skipped-and-named against an external gate.

Reachability: no step here delivers a user-reachable capability until S17 Phase 6 (#314) lands. internal/managementapi/create.go answers 422 to any non-hosted kind, so no API creates a kind=2 container repository — every test and fixture in this plan, Step 18's harness included, seeds repository rows directly. The slice can merge complete and still serve no traffic. This is a scheduling fact for the S16 and S17 DRIs, not a blocker on any step.

Out of scope: S13 owns the format-agnostic foundation (Lookup/Probe/Fetch, the teed cache-fill pipeline, single-flight, cache freshness/fallback, the SSRF-hardened upstream client, credential management, health monitoring). S17 owns remote-repo REST CRUD and the connection test. S32 owns container virtual (kind=1); S29 cache eviction, S28 GC, S20 soft-delete cascade, S08/S09 real auth, S18 the buffered-counter pipeline.

Dependency DAG

graph TD
    ST1["✅ Step 1: schema - repositories/images/manifests (Track A)"]
    ST2["✅ Step 2: schema - relationships/blobs/tags (Track A)"]
    ST3["✅ Step 3: building blocks - URL/segment safety/name transform (Track A)"]
    ST4["✅ Step 4: WWW-Authenticate challenge parser (Track A)"]
    ST5["✅ Step 5: token cache + ContainerRemoteConfig (Track A)"]
    ST6["✅ Step 6: datastore read - finder + health (Track A)"]
    ST7["✅ Step 7: datastore read - cache lookup + freshness (Track A)"]
    ST8["✅ Step 8: datastore write - cache-fill + by-digest gate (Track A)"]
    ST9["✅ Step 9: datastore - credentials/auth_url/auth_status (Track A)"]
    ST10["✅ Step 10: kind dispatch + 405 + 501 stubs (Track A)"]
    ST10_5["✅ Step 10.5: remote handler slots on the dispatcher (Track A)"]
    ST11["✅ Step 11: cache-hit serve + download signals (Track A)"]
    ST12["✅ Step 12: request builder + token-exchange flow (Track B)"]
    ST13["✅ Step 13: operations glue + error mapping (Track B)"]
    ST14["👀 Step 14: manifest proxy + revalidation (Track B)"]
    ST15["👀 Step 15: blob proxy - miss/HEAD/range (Track B)"]
    ST16["✅ Step 16: live lists - tags + referrers (Track B)"]
    ST17["Step 17: observability (Track B)"]
    ST18["Step 18: hermetic e2e proxy harness (Track B)"]
    S13S11["✅ S13 Step 11 part 1: UpstreamDoer adapter + seam assertion"]

    ST1 --> ST2
    ST1 --> ST6
    ST1 --> ST7
    ST2 --> ST7
    ST1 --> ST9
    ST7 --> ST8
    ST6 --> ST10
    ST7 --> ST11
    ST10 --> ST11
    ST10_5 --> ST11

    ST3 --> ST12
    ST4 --> ST12
    ST5 --> ST12
    ST9 --> ST12
    S13S11 --> ST12
    ST8 --> ST13
    ST12 --> ST13
    ST11 --> ST14
    ST13 --> ST14
    ST11 --> ST15
    ST13 --> ST15
    ST10 --> ST16
    ST10_5 --> ST16
    ST12 --> ST16
    ST14 --> ST17
    ST15 --> ST17
    ST16 --> ST17
    ST17 --> ST18

Status

All six schema MRs (1a-1c, 2a-2c) are serialized, not parallel: internal/datastore/migrations/migrations.go sets goose.WithAllowOutofOrder(false) and a migration timestamp is fixed when the file is authored, so each takes the next free timestamp only after the previous one has merged. Both sibling slices serialized theirs for the same reason. Within Step 2 the merge order is 2a → 2c → 2b, relationships last: nothing in this slice reads or writes container_remote_manifest_relationships, so Step 7 waits only on 2a and 2c, and 2b merges off the critical path — it only has to precede Step 8. The plan amendment recording this order rides in the 2a MR.

#

 Description MR Status Done Assignee Dependencies
1a

Schema — container_remote_repositories: root of the chain at 64 hash partitions, interim tmp_plaintext_* credential columns with their all-or-none CHECK, container-only auth_url/auth_status with the CHECK tying them, last_downloaded_at NULLS FIRST retention indexes

!1275 (merged)

workflowcomplete

@radbatnag

 —
1b

Schema — container_remote_images: FK to container_remote_repositories(id, namespace_id)

!1323 (merged)

workflowcomplete

@radbatnag

 1a
1c

Schema — container_remote_manifests: FK to container_remote_images(id, namespace_id), id tiebreaker on both manifest scan indexes

!1354 (merged)

workflowcomplete

@sylviashen

 1b
2a

Schema — container_remote_blobs: drops ADR-007's media_type, adds the created_at it omits

!1430 (merged)

workflowcomplete

@radbatnag

 1b; timestamp order after 1c
2b

Schema — container_remote_manifest_relationships

!1508 (merged)

workflowcomplete

@radbatnag

1b, 1c; timestamp order after 2c — merges last: nothing in this slice reads or writes the table (population deferred to #264); must merge before Step 8 opens, whose acceptance queries it
2c

Schema — container_remote_tags: S13 freshness columns, CR/LF CHECK on upstream_etag

!1503 (merged)

workflowcomplete

@radbatnag

 1b, 1c; timestamp order after 2a
3

Remote building blocks: upstream URL with per-segment path escaping, the outbound reference gate applying S12's tag grammar, and the constant-memory streaming rewrite of the tags/list name field. Fuzz target on the reference gate

!1281 (merged)

workflowcomplete

@radbatnag

 —
4

WWW-Authenticate challenge parser: RFC 7235 quoted strings, case-insensitive Bearer, realm/service extraction, absolute-https requirement, auth_url assembly. Pure functions, fuzz target

!1285 (merged)

workflowcomplete

@sylviashen

 —
5

Token cache + ContainerRemoteConfig: (namespace_id, remote_repository_id, scope) key, TTL arithmetic, single-flight over concurrent misses, per-remote eviction entry point, and the four token-flow durations the cache reads

!1304 (merged)

workflowcomplete

@radbatnag

—; external: S17 (#314, open) supplies the eviction trigger — 3 criteria ship gated
6

Datastore read — remote-repository finder + health: kind=2-scoped finder, HealthStatusReader/HealthStatusWriter over the health columns, RemoteRepositoryEnumerator entries. Touches container_remote_repositories only

!1327 (merged)

workflowcomplete

@sylviashen

 1a
7

Datastore read — cache lookup and freshness: remote.CacheStore.Lookup over the tag, manifest, and blob tables, a container-typed lookup that also returns media_type, and the freshness verdict with DigestKeyed: true

!1558 (merged)

workflowcomplete

@sylviashen

 Steps 1, 2a, 2c — not 2b: the lookup reads the tags, manifests, and blobs tables only
8

Datastore write — cache-fill and the by-digest gate: UpsertCacheEntry writing the image parent plus the manifest or blob row — and the tag row on a tag-reached fill — in one transaction, plus the refusal when CacheEntry.BlobRef disagrees with the requested digest

!1640 (merged), !1641 (merged)

workflowcomplete

@radbatnag

 Step 7; 2b — its acceptance asserts an index fill writes zero relationship rows, a query that needs the table to exist
9

Datastore — credentials, auth_url, auth_status: remote.CredentialRepo over the tmp_plaintext_* columns with the colon rule, the colon validator, both conditional discovery writes and the paired clear

!1346 (merged)

workflowcomplete

@radbatnag

1a; external: S17 (#314, open) owns the transaction, audit event, and API response — 3 criteria ship gated
10

Kind dispatch + write unavailability + 501 read stubs: branch the OCI dispatcher on the kind the resolution already carries, 405 UNSUPPORTED with correct Allow on every write verb, four named stubs in one remote_stub.go, construction-time guard against an unpopulated kind

!1390 (merged)

workflowcomplete

@radbatnag

 Step 6
10.5

Remote handler slots on the dispatcher: one DispatchOption per remote arm — WithRemoteManifest, WithRemoteBlob, WithRemoteTagList, WithRemoteReferrers — each panicking on a nil handler; a slot type carrying the validated namespace.Resolution rather than http.Handler; the slot read in serveRemoteRepository, which becomes a method; and the interim 501 written at each unset branch, retiring all four serveRemote* functions. Steps 11 and 16 need different slots, so neither can own them for the other. Leaves the hosted constructor chain alone (#631)

!1569 (merged)

workflowcomplete

@radbatnag

 —
11

Cache-hit serve path + download signals: ADR-005 delivery mode for blobs, force-streamed manifests, Docker-Content-Digest, Content-Length, Range/206/416, HEAD semantics; owns the shared response-header writer every later remote read path composes, plus the last_downloaded_at bumps

!1694 (merged), !1707 (merged), !1762 (merged), !1764 (merged)

workflowcomplete

@radbatnag

Steps 7, 10, 10.5; external: buffered-counter centralization (#91, open) gates every downloads_count assertion — 3 criteria ship gated, the step does not wait on it, and the last_downloaded_at half is ungated
12

Container request builder + token-exchange flow: VirtualRepositoriesConfig→remote.Config mapping, the RequestBuilder (Accept forwarded, Accept-Encoding: identity, bearer from the token cache), discovery HEAD, exchange GET with Basic auth, auth_status memoization, 404 re-discovery, single retry-on-401, two independent budgets. Writes no UpstreamDoer adapter and edits no internal/remote package

!1456 (merged)

workflowcomplete

@sylviashen

All blockers clear. Steps 3, 4, 5, 9 are merged; the external prerequisite is the UpstreamDoer adapter and the shared remotetest seam assertion from S13 Step 11, both on main via !1290 (merged). Not #269: !1290 (merged) ratified the flat transfer deadline rather than replacing it, and this step writes nothing the eventual fix touches
13

Operations glue + error mapping: per-fetch Lookuper/Fetcher construction, composing Lookup, freshness, Fetch, and CacheFallback, and mapping every remote error and FetchError status onto the OCI envelope

!1784 (merged), !1785 (merged)

workflowcomplete

@sylviashen

Steps 8, 12; external: #397 (closed) gates the plain-gzip half of the identity-representation rejection
14

Manifest proxy — miss fill, digest verification, stale-tag revalidation: Accept forwarded verbatim, both body caps, sha256 checked against a requested digest or adopted as authoritative on a tag; If-None-Match, the 304 bump, the 200 re-point, the null-ETag re-resolve, the pinned-tag skip, cache fallback on transport failure

 !1869 (merged), !1872 (merged), !1873 (merged), !1875 (merged) workflowin review

@radbatnag

Steps 11, 13; external: #433 gap 2 (readable Docker-Content-Digest + pre-commit comparison)
15

Blob proxy — miss fill, streamed 200, cold HEAD, range-on-miss: the 200 teed into the fill; the cold HEAD filling with io.Discard and answering from the committed row; range-on-miss fetching the whole blob with no Range forwarded then serving locally, including fill-then-416

!1894 (merged), !1903 (merged), !1904

workflowin dev

@sylviashen

Steps 11, 13; external: #477 (closed) (single-flight seam shape — 3 coalescing criteria ship gated)
16

Live-proxied lists — tag listing + referrers: one upstream GET with the client's full query string forwarded, Link rewritten to this repository's path, streamed under the large-body cap; wires Step 3's name transform and sets the tag-list and referrers slots

!1547 (merged)

!1716 (merged)

!1717 (merged)

!1718 (merged)

workflowcomplete

@sylviashen

 Steps 10, 10.5, 12
17

Observability — metrics + wide event: the eight oci_remote_* metrics and the oci_remote_proxy wide event, wired at the emission sites Steps 14-16 created

 !1935 (merged) workflowin dev
@radbatnag— Steps 14, 15, 16
18

Hermetic end-to-end proxy harness: one in-package test driving a realistic docker pull sequence through a kind=2 repository against an in-process fake upstream. Deterministic and network-free — no live upstream, no new CI job

 Not started
— Step 17

Cross-workstream dependencies (critical path)

Sibling status verified 2026-08-10. S13 (#280 (closed)) is workflow::in dev. Step 11 (#332 (closed)) is the only S13 step that touches this slice; its Step 12 (#333 (closed)) is virtual resolution, which belongs to S32, and both of its MRs have merged.

  • S13 Step 11 (#332, open, @10io) is three MRs, and the part S16 needs has merged. Step 12 consumes it, every Track B step reaches it through Step 12, and Track A carries none. The split is what unblocks Track B early, because what Step 12 needs is not what is still open: !1290 (merged) (part 1/3) carries the serve taxonomy, the transport marker, the remote.UpstreamDoer adapter over upstreamhttp.Client (internal/remote/upstreamhttp/doer.go), and the shared remotetest.AssertUpstreamDoerConformance seam assertion — that adapter and that assertion are the whole of Step 12's external need, and both are on main. !1291 (merged) (part 2/3, the standalone request flow) is open with unresolved discussions, and !1292 (merged) (part 3/3, its choreography tests) is draft and conflicted. Neither delivers anything this slice consumes: the standalone flow is a different entry point, which the plan records in Step 13's Note. S16 still writes no adapter and edits no package under internal/remote.

  • #269** is open again, so Step 11 no longer carries a transfer-deadline fix.** A Closes trailer on !1199 (merged) — S16's own plan MR — closed it on 2026-08-03; @10io reopened it on 2026-08-05 because the close was a side effect, not a decision that the work was done. !1290 (merged) instead records the gap and defers the fix: the interim behavior stays Client.Do's flat request_total_timeout, and the progress-based read deadline that would tell a stalled transfer from a slow-but-moving one is still to come. No S16 criterion depends on it, but that flat cap is the real ceiling on how large a layer Step 15 can fill over a slow upstream. Of the two questions @10io left on #269, one is answered: docs/dev/configuration-reference.md carries the operator sizing paragraph, which landed with !1290 (merged) and was sharpened by !1328 (merged). The other still wants an answer before Track B starts — what request_total_timeout GitLab.com actually runs, which needs a Vault lookup. Of #269's four proposed fixes, 2 and 4 landed as S13 spec amendments in !1329 (merged) — fix 2 confirming a reading that is S32-owned, fix 4 recording the blob-cap reachability gap rather than closing it. Fixes 1 (the progress-based read deadline) and 3 (cache fill survives client disconnect) are code in internal/remote, unassigned, and have no S13 plan step left to carry them, so each needs an owner plus a spec MR and plan MR of its own. The S16 plan repeated the superseded claim that Step 11 resolves #269 — including in Step 12's own prerequisite bullet — and !1397 (merged) corrects it.

  • Fetch coalescing is deferred by decision, not by a missing dependency. S13's single-flight layer (#329 (closed)) is merged, but NewSingleFlight holds the RequestBuilder and CacheStore for the repository's lifetime, while both seams are request-scoped here on purpose — the container builder holds a bearer token it must not outlive, and the cache store needs the request's digest for Step 8's gate. Reconciling the two is #477 (closed); Steps 13 and 15 keep the request-scoped seams and ship 3 criteria skipped against it. Cross-instance coordination is #375 and S16 asserts nothing that needs it. Two things moved since: the builder-lifetime half now has a spec-side fix queued as #320 (closed) item 9 — an implementation reads credentials at call time inside BuildRequest rather than capturing them at construction — and #528 proposes buffering the first body chunk inside remote.SingleFlight so a body that dies before its first byte can still fall back to a stale cache row. #528 is deferred by operator decision, and as designed its fix lives inside SingleFlight, which S16 does not compose — Step 14's cache fallback would keep the same gap.

  • S13 health monitoring is complete. Step 14 (#335 (closed)) and Step 15, the scheduled sweep (#336 (closed), 2026-08-06), are both merged, so the probe, status transitions, failure counter, on-demand entry point, and the periodic sweep in internal/remote/healthjob.go are all on main. Step 6 waits on none of it, and both probe criteria stay verifiable against remotetest.FakeHealthStatusStore. The live successor is #479 — one HEAD per distinct upstream instead of one per row — which also proposes revisiting the RemoteRepositoryEnumerator shape (pagination or streaming) in the same design pass. Step 6 delivers this format's enumerator entries against today's shape; if #479 lands first, Step 6 follows whatever it settles.

  • S13 fetch-seam gaps (#433, open, unassigned) — gap 2 (readable Docker-Content-Digest plus a pre-commit comparison) gates Step 14's disagreeing-header criterion in full; gap 3 (Content-Length exemption on the forwardable-header allowlist) gates the forwarding half of Step 15's cold-fill criterion. Both shared with S14. Both are downstream of the safeHeadersFrom allowlist that gaps 2 and 3 have to extend, which !1290 (merged) landed on main. One allowlist cannot express the outcome-plus-raw gate those two gaps need, so the shape has to be settled against the merged safeHeadersFrom. Gap 1 is in flight as !1325 (closed) (draft).

  • S13 per-request no-transparent-decode switch (#397 (closed), open, @10io, quiet since 2026-07-31) — gates the plain-gzip half of Step 13's identity-representation rejection. Shared with S14, whose merged upstream builder carries the same gap.

  • S13 spec amendment (#320 (closed), open) — item 8 is auth_status in the behavioral-columns table and the clear-on-url-change rule. Spec text, not code; until it lands, S16's spec is the only place the column and its reset are written down. The issue has grown to 11 items and two of the newer ones also reach this slice: item 9 above, and item 11, which excludes the read-time size caps (ErrResponseSizeTooLarge, ErrDecompressionTooLarge) from the cache-fallback family because they are raised during a body Read, never returned from UpstreamDoer.Do, and so never carry the transport marker. Item 11 was settled in the !1290 (merged) review and its code-side doc fix is on main; Step 13's error mapping follows it.

  • S17 Phase 6 remote-repo CRUD + connection test (#314, open, unassigned, no activity since 2026-07-15) — four auth criteria assert its write surface (Step 9 ships the store half of each), it is the trigger for Step 5's three eviction criteria, and it owns the single transaction around a url change. Also the gate on this slice being user-reachable at all. Nothing has moved on it over the whole Track A run so far.

  • Buffered-counter centralization (#91, open, unassigned) — gates three downloads_count assertions in Step 11 and nothing else in this slice. Step 11 itself is not blocked by it. The spec and the plan both already say how to ship without it: wire each read to the store seam carrying the increment, and ship the store method as a stub that builds the UPDATE and returns nil without executing it, matching MavenRepositoryStore.BumpRepoDownloadCount in internal/datastore/maven_repositories.go. The last_downloaded_at bumps are ungated and must work; npm's injected-bumper shape is available to follow.

    This issue previously called #91 "the S18 buffered-counter pipeline", which overstates it. #91's scope is moving the stub helpers into one shared package so Maven, npm, and OCI stop each carrying their own, and its exit criteria keep the stub as "currently direct synchronous UPDATEs". Whether the shared helper executes the downloads_count increment is not settled anywhere in #91's text, so #91 merging does not on its own turn those three assertions on — it moves the call site, which is not the same as making the counter count. The working bounded writer that would do the counting is npm's (internal/format/npm/buffered.go), format-private and out of reach per ADR-023, and putting it within reach is what #91 would change. Whoever picks up #91 has to decide the executing-versus-inert question explicitly; until then, treat the gate as still closed however far #91 gets.

    Still unassigned, still no spec, and no shared package exists (internal/counters/ or otherwise). Moved on 2026-08-14 for the first time since 2026-06-10: @dmeshcharakou opened #632 — no buffered counter column has a reconciliation pass, so a write that is shed drifts permanently — and linked it to #91 as the durability half that centralizing on its own does not address. #632 does not reach this slice: Step 11's only buffered write is the inert downloads_count stub, and the last_downloaded_at bumps it does execute are timestamps a later read repairs rather than counters that accumulate drift.

  • S12 delivery-mode questions (#434) — whether a 307 carries Docker-Content-Digest, and how Range interacts with redirect delivery. Nothing blocked: the affected criteria are scoped to proxied responses.

  • ADR-007 reconciliation (#30, items 6-11) — a handbook-repo MR, not a step here, and no open handbook MR carries these items yet. The container range grew by one since this issue was written: item 11 records CHECK (last_health_status IN (0, 1, 2)) on the container block, with the npm and Maven halves split out to #486 (closed). Until the amendment lands, a reviewer diffing Steps 1-2 against ADR-007 will flag correct migrations as divergent, auth_status most visibly.

  • Namespace partition key on the health and credential seams (#445 (closed)) — merged, so Steps 6 and 9 implement the already-widened signatures. Taking namespace_id is not the same as pinning it in the predicate, and only the pinning prunes the 64 partitions; each step carries a pinning note.

  • Credential encryption at rest (#417) and the realm-host allowlist (#263) stay GA-blocking follow-ups. Step 1a ships interim tmp_plaintext_* columns.

Related

Edited by Radamanthus Batnag