Container remote and virtual: deferred follow-ups (S16, S32)

Context

Deferred follow-ups from the container remote and container virtual specs — S16 (MR !797 (merged)) and S32 (MR !972 (merged)). These are out of scope for the closed-beta MVP; each needs a design before implementation. Confirm scope before starting any item.

Both specs point their Follow-ups entries at this work item, so the list is split by owning spec below. An item's owner is the spec whose text has to change for it, not the spec that first noticed it.

S32 — container virtual

  • Cross-upstream tag listing. GET .../tags/list on a virtual container repository answers 404 NAME_UNKNOWN and runs no resolution. S13's model resolves one path across upstreams and has no operation for merging a paginated collection across N upstreams with independent cursors; designing that merge and its cursor scheme is deferred. S16 serving the endpoint on a single-upstream remote does not supply it — one upstream's cursor is exactly what N upstreams cannot share, and a hosted winner has no upstream to proxy.
  • Cross-upstream referrers. GET .../referrers/<digest> answers 404 NAME_UNKNOWN for the same reason: no cross-upstream merge of subject_digest matches. The 404 is deliberate rather than incidental, because it lets an OCI client fall back to the referrers tag schema, which does resolve across upstreams. Deferred pending the merge design.
  • ADR-007 amendment for the container virtual schema. Four details in S32's Data Model are not in ADR-007 and need a handbook MR, which cannot be raised from this repository: the (namespace_id, upstream_repository_id) reverse index on container_virtual_repository_upstreams; the ON DELETE action on every foreign key, which ADR-007 leaves unstated; the range and length CHECKs on rule_type, target_field, pattern, and position, which ADR-007 gives as prose value legends only; and the correction of DEFERRABLE INITIALLY DEFERRED from an index clause to a constraint clause, since deferrability is a property of a constraint and CREATE UNIQUE INDEX ... DEFERRABLE is a syntax error. The Maven and npm virtual tables carry the same wording, so the amendment covers all three formats.
  • ADR-009 amendment for the two virtual read routes. ADR-009 annotates only write routes as unavailable on remote and virtual repositories, so GET .../tags/list and GET .../referrers/<digest> carry no annotation and S32 records its 404 as a deliberate divergence. Annotating both routes retires the divergence. Do this alongside the merge design above, or ahead of it if the 404 is to stand for longer.
  • Resolution-phase coalescing. S13's single-flight covers Fetch, so concurrent cold reads of one path produce one upstream GET, but Phase 2 probes are not coalesced and resolution.probe_concurrency bounds one resolution rather than the process. A cold-cache image pull across C concurrent clients against M remote upstreams can issue up to C×M HEAD requests plus their token exchanges, and an upstream that rate-limits under that load is health-marked unhealthy, which takes every image behind it to a 503. The MVP accepts this: the probe is a bodyless HEAD, the token exchange is already single-flighted per upstream and scope, and ADR-004 caps upstreams at 20. gitlab_artifact_registry_oci_virtual_probes_total is the signal that says whether it needs closing, so this item is gated on that metric rather than on judgement. Closing it needs a resolution-level single-flight key, which S13 does not define today.

S16 — container remote

  • Ranged Fetch on a cache miss. The MVP fetches and caches the full blob first, then serves the range from local storage. Forwarding the client Range upstream and streaming it through without buffering the whole blob, and without caching a partial 206, is a deferred optimization. A virtual read inherits this behavior through its remote winner rather than having a say in it.
  • Lazy manifest-relationship population and progressive size. Lazy caching stores a parent manifest index before its children exist, so the MVP writes no parent-child links to container_remote_manifest_relationships and a manifest-list or index size stays at its own payload length. Defining the link-on-child-arrival mechanism and the progressive-size convergence is deferred. This table is S16's; S32's data model has no equivalent.

Not tracked here

Container Probe timing and the large-layer transfer gaps are S13-owned and tracked in #269. S32 discharges that issue's probe_timeout item for the container path by charging the token handshake to container_remote.token_exchange_timeout and the HEAD leg to resolution.probe_timeout, which S13 now fixes as bounding the HEAD alone — so no S13 amendment is owed for the Probe. The body_size_cap_blob reconciliation against the 50 GB hosted cap stays open there.

Edited by Radamanthus Batnag