test(oci): hermetic end-to-end proxy harness (S16 plan: 18/26)

One in-package harness driving a realistic docker pull — the manifest by tag, then the config blob and each layer blob it names — through a single kind=2 container repository against an in-process TLS httptest fake registry. This is S16's conformance deliverable for a proxy path, as Step 18 is in S14.

It composes the dispatcher, both cached read arms, a real remote.Fetcher and the real security-hardened upstream client over three in-memory seams: the container_remote_repositories row, the four container_remote_* cache tables, and internal/storage/stub for object storage. The assembly mirrors (*ociRemoteFill).operationsFor in cmd/artifact-registry/wire_oci_remote.go. Deterministic and network-free: no live upstream, no Postgres, no new CI job, and unlike the Maven and npm harnesses no integration build tag.

What it establishes

Seven claims. The first four are properties of a sequence rather than of one request, so no per-arm suite can assert them. The last three are properties of the composed stack:

• a cold pull fills the cache and serves the artifact, at one discovery HEAD and one GET per document;
• a repeat pull issues zero upstream calls and serves identical bytes;
• a stale tag revalidates on a 304, serves the manifest the cache already held, writes no new row, and bumps upstream_checked_at;
• a token-auth upstream completes the handshake once — one discovery HEAD and one token exchange — across four artifact reads;
• a second kind=2 repository over the same upstream image fetches the manifest again rather than reading the first repository's cached rows;
• a manifest PUT through the composed dispatcher is refused 405 UNSUPPORTED with Allow: GET, HEAD, at no cost to the upstream;
• an upstream 403 past a valid token reaches the client as this service's own DENIED envelope, naming neither the token nor the upstream authority.

The cache tables key every row by repository and path together, and the factory each arm resolves through hands out a per-repository store. That is what datastore.NewContainerRemoteCacheStore's repo argument buys in production. A map keyed on the path alone cannot stand in for it, because the read it admits — one repository answered from another's cached rows — is the highest-consequence bug a proxy cache has, and no assertion could see it.

Three assertions are falsifiable that were not. The last two were checked by running the mutation and watching them fail:

• the fake blob route answers a Docker-Content-Digest that disagrees with the blob it serves, so a relayed header now fails where an echoed one passed either way. The manifest route keeps its caveat, being unfalsifiable here for the same reason;
• the upstream 403 is the first error envelope this file produces, and it carries WWW-Authenticate plus the token and the upstream authority in its body. Asserting that WWW-Authenticate reaches the client fails there; on a 200 no such response exists to assert against;
• pointing the sibling repository's store at the first repository's id drops the upstream GET count to one and fires the cross-repository assertion.

Upstream calls are matched on method and target together. Counting HEADs by URL or GETs by host each passes against a broken implementation, and the host-scoped shortcut happens to read as correct against a realm sharing the upstream's host, which this fixture's does. Calls also carry their query string, so the one token exchange is checked for what it asked for: service is the value the challenge named, and scope is repository:library/nginx:pull — pull only, with the multi-segment image name intact.

Six tests pass and six skip:

--- PASS: TestRemoteProxy_E2EDockerPullSequence
--- PASS: TestRemoteProxy_E2EStaleTagRevalidatesOn304
--- PASS: TestRemoteProxy_E2ETokenAuthUpstreamHandshakesOnce
--- PASS: TestRemoteProxy_E2ECachedRowsDoNotCrossRepositories
--- PASS: TestRemoteProxy_E2EWriteVerbIsRefused
--- PASS: TestRemoteProxy_E2EForbiddenUpstreamLeaksNothing
--- SKIP: TestRemoteProxy_E2EDisagreeingDigestHeaderDiscardsTheFill
--- SKIP: TestRemoteProxy_E2ECoalescesRangedAndFullBlobMiss
--- SKIP: TestRemoteProxy_E2ECoalescesHEADAndGET
--- SKIP: TestRemoteProxy_E2ECoalescedFollowerReadsTheCommittedRow
--- SKIP: TestRemoteProxy_E2EPullRecordsADownload
--- SKIP: TestRemoteProxy_E2EScheduledSweepProbesThisRepository

The gated ledger

The step's second deliverable is the ledger that outlives the plan: remoteE2EGatedLedger maps each skipped criterion to the work items whose landing lifts it, and remoteE2ESkip fails on an unknown row id so a rename cannot quietly un-gate a test. Each skip prints its items, so go test -run Remote -v is the outstanding list rather than a merged plan nobody reopens.

The ledger names four items where the plan's Step 18 entry named six, and every difference is the plan being stale rather than the harness being wrong:

Plan named	Ledger	Why
#336 (closed)	absent	Closed, and buildContainerHealthSweepSource is one of the builders entries in cmd/artifact-registry/wire_remote.go, so a sweep reaches container remotes and the coverage lives there
#314	absent	Its create gate has lifted — internal/managementapi/create.go accepts kind=remote
#433	#918	#433 is open and covers S13's fetch-seam gaps, the readable Docker-Content-Digest among them; the container-side cross-check this criterion needs is #918, because commitUpstreamManifest passes no expected digest

Guardrail 4 bars a step MR from editing the plan, so those corrections land on the standing S16 catcher, !2077 (merged), which also carries the matching spec fix: the criterion's blocker text said settleUncachedRead answers 501 so no fill exists, and it fills now.

One row carries no work item — the scheduled health sweep, covered where it is wired — and remoteE2ESkip prints no un-skipper for it rather than pointing at something that will never lift it.

Merge order

This MR targets main and depends on nothing. It was stacked on rbatnag/container-remote-step-17-2 while !2078 (merged) was open; that branch is now gone, so review the diff against main. Nothing here gates !2077 (merged), and !2077 (merged) gates nothing here.

Reviewable LOC

Past the 500-LOC bar development-model.md sets, so the justification, split by file group. Measured at 7332e034c, against the merge base:

Group	LOC
internal/format/oci/remote_e2e_test.go	+1787
docs/testing/e2e/docker.md	+45 −6
docs/testing/e2e/oci.md	+15 −10

Splitting does not help. There is one Go file and it is one harness: the fixture, the fake registry, the four cache-table fakes and the storage seam are what the three passing tests all drive, and no subset of them stands up a test on its own. It is also the plan's estimate that was wrong rather than the harness that is oversized — the entry forecast ~580 lines and did not account for the fake registry needing its own token endpoint, challenge, and per-call method-and-target recording.

e2e scenarios

Both catalogs are updated, so the revisit guardrail is satisfied rather than waived. docs/testing/e2e/docker.md gains a ## Remote repositories section with five docker-driven rows and drops remote from Out of scope; docs/testing/e2e/oci.md gains four HTTP rows for the manifest arm and corrects its arms-mounted statement, all four read arms now being wired.

Every row's Status stays not started, this harness notwithstanding. Those rows name the out-of-repo suite that drives docker, oras and raw HTTP at a deployed instance; an in-package harness does not start one.

Spec coverage

Spec: docs/specs/S16-container-remote.md, section "API Contracts".

Step 18 adds no production code and no unit suite. Its deliverable is one harness, so most entries are owned by the per-step suite that shipped the arm; those name the suite's file rather than a test, because the entry is a contract the whole file pins rather than one case. E2E names a test in internal/format/oci/remote_e2e_test.go. Rendered flat rather than as a markdown table, matching the shape the commit body carries.

API contracts

Each entry is an id, its criterion, then the tests covering it.

AC-1 /v2/ GET unchanged (S08/S09 stub) Owned by S08/S09. Not exercised here.

AC-2 manifests/<reference> GET, HEAD backed by the manifest proxy E2E (GET); remote_serve_test.go, remote_manifest_test.go

AC-3 manifests/<reference> PUT, DELETE → 405 UNSUPPORTED E2E (PUT, through the composed dispatcher); remote_dispatch_test.go

AC-4 blobs/<digest> GET, HEAD backed by the blob proxy E2E (GET); remote_blob_serve_test.go, remote_blob_test.go

AC-5 blobs/<digest> DELETE → 405 UNSUPPORTED remote_dispatch_test.go

AC-6 blobs/uploads/* every verb → 405 with a present-but-empty Allow remote_dispatch_test.go, remote_slots_test.go

AC-7 tags/list GET proxies live remote_taglist_test.go, remote_list_handler_test.go

AC-8 referrers/<digest> GET proxies live remote_referrers_test.go, remote_list_handler_test.go

AC-9 No AR-side pagination validation on the live lists; query forwarded verbatim remote_taglist_test.go, remote_referrers_test.go

AC-10 Query string over 8 KiB → 400 QUERY_LIMIT_EXCEEDED before any upstream call remote_list_handler_test.go, remote_taglist_test.go

AC-11 Range on a client HEAD ignored: 200, never 206/Content- Range/416 remote_blob_serve_range_test.go

AC-12 HEAD resolves through the same Lookup/Fetch path and fills on a miss remote_blob_test.go, remote_manifest_test.go. The harness drives GETs only.

AC-13 The proxy read builds its own headers and never blanket-copies the upstream's E2E, via remoteE2EAssertNoUpstreamHeaders on both the 200s and the upstream-403 envelope; remote_serve_headers_test.go. On the two routes this harness drives, the property holds by signature: writeRemoteReadHeaders takes no upstream response, so it cannot relay one. The sweep's Set-Cookie, X-Upstream-Only and ETag arms therefore guard a refactor that passes one in, which is a narrower claim than the criterion and worth keeping on its own terms.

AC-14 Relay allowlist applies to the two live lists only remote_relay_internal_test.go, remote_taglist_test.go, remote_referrers_test.go

AC-15 Docker-Content-Digest service-generated on every manifest and blob read E2E (tag read and by-digest blob read); remote_serve_headers_test.go. The blob half is falsifiable: the fake answers a disagreeing header, so a relayed value fails. The manifest half is not, and its assertion carries that caveat.

AC-16 Content-Type: manifest from the stored media_type, blob application/octet-stream E2E; remote_serve_test.go, remote_blob_serve_test.go

AC-17 Content-Length is the stored payload's length; cache-miss blob GET forwards the upstream's or none E2E (cached manifest); remote_serve_headers_test.go, remote_blob_test.go (cold-fill subset)

AC-18 Content-Range on a 206/416, computed against the cached blob's length remote_blob_serve_range_test.go

AC-19 nosniff on every response; Content-Disposition: attachment on a blob 200/206 E2E; remote_blob_serve_test.go, remote_blob_serve_range_test.go

AC-20 Retry-After on every 503 and on a propagated 429, validated and clamped remote_errors_test.go, remote_blob_serve_test.go

AC-21 Allow on a 405 where the route has a read verb to advertise E2E (manifest PUT); remote_dispatch_test.go

AC-22 Per-endpoint additions: Link (rewritten), OCI-Filters-Applied remote_taglist_test.go, remote_referrers_test.go, remote_relay_internal_test.go

Error cases

Numbered in the spec's own row order.

Each entry is an id, its criterion, then the tests covering it.

E-1 Manifest or blob write verb → 405 + Allow: GET, HEAD E2E (manifest PUT); remote_dispatch_test.go

E-2 Any verb on a blob-upload route → 405 + empty Allow remote_dispatch_test.go, remote_slots_test.go

E-3 Read before that route's proxy arm lands → 501 INTERNAL remote_slots_test.go

E-4 Read whose arm landed but whose cache cannot answer → 501 INTERNAL remote_serve_test.go, remote_blob_serve_test.go

E-5 Any verb on a kind=1 repository → 501 INTERNAL remote_dispatch_test.go

E-6 kind outside {0,1,2} → 503, or 500 with no authz middleware remote_dispatch_test.go, authz_composition_test.go

E-7 kind=2 with no container_remote_repositories row → 500 INTERNAL remote_serve_test.go, remote_blob_serve_test.go

E-8 Discovery HEAD establishing no verdict → no write, acquisition fails remote_auth_test.go

E-9 Challenge missing realm or service → proceed with no token remote_challenge_test.go, remote_auth_test.go

E-10 realm not an absolute https URL → cache-or-503 remote_challenge_test.go, remote_auth_test.go

E-11 Token-exchange transport failure → cache fallback, else 503 remote_auth_test.go, remote_errors_test.go

E-12 Half-set credential pair → 500 INTERNAL, no upstream request remote_auth_test.go, remote_credentials_test.go

E-13 Token exchange 404 on a cached auth_url → clear and re-discover once remote_auth_test.go

E-14 Token exchange 401/403 → propagated, no cache fallback remote_auth_test.go, remote_errors_test.go

E-15 Token exchange other non-2xx → cache fallback, else 503 remote_auth_test.go, remote_errors_test.go

E-16 Upstream 401 after a fresh token → 401 UNAUTHORIZED remote_errors_test.go, remote_tokencache_test.go

E-17 Upstream 403 → 403 DENIED E2E; remote_errors_test.go

E-18 Upstream 5xx → propagated verbatim, code by status remote_errors_test.go

E-19 Unenumerated upstream 4xx → propagated verbatim, INTERNAL remote_errors_test.go

E-20 Upstream 405/407/426 on a manifest or blob read → 503 UNAVAILABLE remote_errors_test.go

E-21 Upstream 404 for a manifest reference → 404 MANIFEST_UNKNOWN remote_manifest_test.go, remote_errors_test.go

E-22 Upstream 404 for a blob digest → 404 BLOB_UNKNOWN remote_blob_test.go, remote_errors_test.go

E-23 Digest mismatch before headers commit → 503, nothing cached remote_cache_store_test.go, remote_manifest_test.go

E-24 Digest mismatch on a streamed blob GET → interrupted 200 remote_blob_test.go

E-25 List query string over 8 KiB → 400 QUERY_LIMIT_EXCEEDED remote_list_handler_test.go, remote_taglist_test.go

E-26 Upstream non-2xx on a list → propagated, code by status remote_taglist_test.go, remote_referrers_test.go

E-27 Upstream 405/407/426 on a list read → 503 UNAVAILABLE remote_list_internal_test.go

E-28 Upstream 429 → propagated with Retry-After remote_errors_test.go

E-29 Tag resolution whose Docker-Content-Digest disagrees → 503 Gated on #918. The E2E test for it skips, naming the item.

E-30 Manifest payload the detection chain refuses (four shapes) → 503 remote_cache_store_test.go, mediatype_test.go

E-31 Manifest payload that does not decode into the envelope → 503 remote_cache_store_test.go

E-32 Manifest read-back fault (three shapes) → 500 INTERNAL remote_cache_store_test.go

E-33 Digest reference fails format validation → 400 DIGEST_INVALID remote_serve_test.go, remote_blob_serve_test.go

E-34 Non-digest reference fails the tag grammar → 404 MANIFEST_UNKNOWN remote_serve_test.go

E-35 Content negotiation mismatch → 404 MANIFEST_UNKNOWN remote_serve_test.go

E-36 Transport failure, no cache entry → 503 UNAVAILABLE remote_errors_test.go, remote_manifest_test.go

E-37 Read against a durably unhealthy remote → 503 UNAVAILABLE remote_manifest_test.go, remote_blob_serve_test.go

E-38 Coalesced read whose single-flight wait timed out → 503 Gated on #752. No single-flight layer is composed; the E2E test skips, naming the item.

E-39 Transport failure with a cache entry → 200 from cache remote_revalidate_test.go, remote_errors_test.go

E-40 Upstream body over the configured cap → 503 remote_cache_store_test.go, remote_operations_test.go

E-41 Undecodable content coding despite Accept-Encoding: identity → 503 Owned by internal/remote/upstreamhttp. Not re-asserted here.

E-42 <image_name> fails the grammar, or the path exceeds 255 → 400 NAME_INVALID remote_dispatch_test.go, remote_upstream_test.go

E-43 Slug or repository does not resolve → 404 NAME_UNKNOWN remote_dispatch_test.go

E-44 Unsatisfiable Range on a blob GET → 416 + Content-Range: bytes */<n> remote_blob_serve_range_test.go

Security considerations

Each entry is an id, its criterion, then the tests covering it.

S-1 Token and credential hygiene: never in logs, metrics, traces or responses remote_auth_test.go, remote_tokencache_test.go, remote_wideevent_test.go. E2E additionally pins that an upstream 403 carrying WWW-Authenticate, the token and the upstream authority reaches the client as a DENIED envelope naming none of the three.

S-2 Auth-challenge trust: the realm URL runs the upstream client's security covers remote_challenge_test.go, remote_auth_test.go, internal/remote/upstreamhttp. The realm-host allowlist is a GA- blocking follow-up (#263).

S-3 Cross-origin redirect credential stripping Owned by internal/remote/upstreamhttp. Not re-asserted here.

S-4 Outbound path-segment safety: every interpolated segment validated first remote_upstream_test.go, FuzzOutboundUpstreamURL. E2E's multi- segment image name additionally pins that a valid / stays segment- separated outbound.

S-5 No amplification via retries: the 401 retry is bounded to one per request remote_auth_test.go, remote_tokencache_test.go

S-6 Upstream auth failures do not leak into client-facing auth remote_errors_test.go; E2E drives an upstream 403 past a valid token and pins that the upstream's WWW-Authenticate is absent from the client response.

Related to #288