## Summary Follow-ups from the local e2e pass on the Maven remote metadata fill ([!1745](https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1745)), plus a later read of [S14](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/specs/S14-maven-remote.md) against how a typical Maven remote (Nexus, Artifactory, a careful Central mirror) behaves. The fill matches S14. These are not bugs in that MR. They are places the spec is stricter or more conservative than those remotes, and they are worth a spec decision later. ⚠️ Do not fold any of them into !1745. --- ## Checked against the code (this branch / !1745) Read of the live handlers, not only S14. No drift that kills a point. | # | Claim | Code | |---|---|---| | 1 | Cold metadata GET has no ETag / checksums | `streamMetadataFill` in `remote_metadata.go` sets Content-Type, Disposition, optional Content-Length only | | 2 | Metadata window cannot be 0 | `CHECK (metadata_cache_validity_hours > 0)` in the maven remote migration | | 3 | Upstream 5xx does not serve cache | `mapFetchErrorStatus` in `remote_errors.go`: 500 and other 5xx are `RespondError`; only transport / unusable-status uses `cacheFallbackOrUnavailable` | | 4 | HEAD miss fills the whole file | `dispatchArtifactFill` / `drainArtifactFill`: HEAD miss drains the streamed GET into cache, then serves headers | | 5 | 404 writes no row | `TestRemoteMetadata_Miss_Upstream404` and the artifact twin assert no cache row | | 6 | Revalidation is ETag-only | Artifact/metadata fill set `IfNoneMatch` from `res.ETag` only. No `If-Modified-Since` in `internal/format/maven`. S13 `SafeHeaders` may *forward* Last-Modified; Maven does not *store* it or send IMS | | 7 | Group plugin path is a fake package | S10 parse: last segment = artifact_id. No special case in `remote_metadata.go` | | 8 | Hits may 302 | Remote hits call `servePrimaryGet` → hosted `download.go` `serveRedirect` when S06 chooses redirect. Local e2e forced `DELIVERY_MODE_PROXY`, so we did not observe a 302 on this box | --- ## 1. 📦 Send checksums and ETag on a metadata cold fill ### What happens today A cache-miss `GET` for `maven-metadata.xml` streams the body before the tee finishes. The first `200` has `Content-Type` and `Content-Disposition`, but no `ETag` and no checksum headers. The next request for the same path has the full header set. S14 states this on purpose. The same rule covers jars (up to 5 GB). You cannot wait to hash a multi-gigabyte jar before you start the response. ### Why metadata is different `maven-metadata.xml` is small. Often a few KB. The cap is 16 MB. Maven Resolver checks the file against a checksum, usually `.sha1`. When the first response has no checksum header, Resolver makes a second request for `maven-metadata.xml.sha1`, then compares. One metadata fetch becomes two HTTP calls on every cold fill and every revalidation `200`. ### Suggested change For metadata only: 1. Fetch the XML. 2. Hash it. 3. Commit the cache row. 4. Then send `200` with `ETag` and checksum headers, plus the body. Jars keep the current stream-first behaviour. This needs a spec amendment first. The Step 15 tests pin the current cold-fill header subset. --- ## 2. 🔄 Package-level metadata cannot be “always check” ### What happens today Package-level metadata is `com/example/demo/maven-metadata.xml`. It is the version list. CI uses it to find the latest release. S14 rejects `metadata_cache_validity_hours = 0`. The shortest window is 1 hour. `snapshot_metadata_always_revalidate` only applies to the version-level file under a `-SNAPSHOT` directory. The package-level file still uses the window. There is no purge-this-path API on this fill. ### Why that hurts You publish `1.0.1`. A client that hit our remote 10 minutes ago still gets the old version list until the window ends. The file people care about most after a release is the one we cannot put in “check every time.” The spec chose this so a typo cannot turn every metadata GET into an upstream GET, and so a version is never hidden forever. The snapshot opt-in exists because snapshot *builds* change many times a day. The version *list* was treated as slower. Typical remotes let you set a very short metadata TTL, allow 0, or invalidate one path after a release. ### Suggested change (product call) Pick one, in a spec amendment: - Allow `metadata_cache_validity_hours = 0` (“never serve package-level metadata from age alone”). - Or extend always-revalidate to package-level metadata. - Or add a path-level invalidate once a management API exists. Do not invent a third mode in the fill handler without that amendment. --- ## 3. 🧯 Upstream 5xx does not serve stale metadata ### What happens today If the upstream is unreachable (timeout, connection error), and we already have a cached `maven-metadata.xml`, we serve that copy and log served-from-cache. If the upstream answers `502` or `503`, we do **not** serve the cache. We map the status (`500` → `internal_server_error`, other `5xx` → `upstream_unavailable`) and fail the client. A cached copy stays unused. That split is S14 error mapping, shared with artifacts: a `5xx` is “the origin spoke and it is sick,” not “we could not talk to it.” ### Why metadata is different For a jar, refusing to serve a stale file when Nexus is returning 503 is reasonable. You do not want to hand out a known-old binary and call it fine. For the version list, CI usually prefers yesterday’s `maven-metadata.xml` over a hard fail. A 503 on that file blocks every resolve that needs “what versions exist?”, even when we have a perfectly usable list from an hour ago. Many CDNs call this stale-if-error, and they turn it on for this kind of document. ### Suggested change (product call) In a spec amendment, decide whether a metadata (not artifact) revalidation or miss that sees upstream `5xx` may serve the cached document when one exists, the same way a transport failure already does. Do not change the artifact path as part of that decision. A sick origin on a jar is a different risk than a sick origin on a version list. --- ## 4. 📡 A HEAD miss can pull a whole jar ### What happens today A `HEAD` for a file we do not have does the same upstream `GET` and cache fill as a real download, then returns headers only. For metadata that is cheap. For a jar it can be up to 5 GB. S14 does this so the `HEAD` can return `ETag` and checksums. Those only exist after a full fill. ### Why that hurts IDEs, Dependabot-style scanners, and some Gradle/Maven plugins `HEAD` many paths they will never download. One probe of a fat jar is a full Central fetch and a full CAS write. A typical Maven remote `HEAD`s the upstream (or uses a tiny range), returns length / type, and does not store 5 GB because someone probed. ### Suggested change (product call) In a spec amendment, decide whether a remote `HEAD` miss for an **artifact** may probe the upstream without a full fill, and keep the current “fill then reply” behaviour for metadata (small) and for `GET`. This is the largest origin-load gap that is not about the metadata window. --- ## 5. 🕳️ A 404 is not remembered ### What happens today The upstream says the path is missing. We return `404` to the client. We store nothing. The next resolve of the same missing path hits the upstream again. ### Why that hurts Optional dependencies and “try this classifier, then fall back” generate many 404s. A clean CI graph can hammer Central on the same missing files every build. A typical remote keeps a short “this path is gone” record (often 1–15 minutes). ### Suggested change (product call) In a spec amendment, decide whether a miss `404` may write a short-lived negative cache row (or equivalent), with a TTL separate from the content windows, and never treat that row as a successful fill for sidecars or counters. --- ## 6. 🏷️ Revalidation is ETag-only ### What happens today A stale row replays `If-None-Match` when `upstream_etag` is set. There is no `Last-Modified` / `If-Modified-Since`. If the upstream sent no ETag (some Nexus and GitHub Packages layouts), every stale check is a full re-download. ### Suggested change (product call) Store `Last-Modified` when the origin sends it, and send `If-Modified-Since` when there is no ETag. Keep ETag as the first choice when both exist. --- ## 7. 🧩 Group plugin metadata is stored as a fake package ### What happens today `org/apache/maven/plugins/maven-metadata.xml` is a real Maven request (plugin prefix resolution). S14 already says it is cached as `group_id=org.apache.maven`, `artifact_id=plugins`. The bytes are correct. The row is not a real package. S14 parks a proper cache identity for a future format spec. ### Why that still matters The first Maven UI, search, or “list packages in this remote” will show junk coordinates. Retention will treat `plugins` as a package. ### Suggested change Keep the current proxy behaviour (bytes and upstream URL are right). When a later format spec gives group-level metadata its own cache key, migrate or stop creating these fake package rows. Do not invent that key in a fill MR. --- ## 8. ↗️ Cache-hit delivery may redirect ### What happens today On a cache **hit**, S06 may `302` to object storage (ADR-005). Local e2e forced proxy mode. Production may not. ### Why Maven is picky Maven Resolver is uneven with signed S3 URLs (query strings, redirects, some corporate proxies). Most Maven-focused remotes stream the jar themselves. Redirect is fine for OCI or a browser download; it is a sharp edge for `mvn`. ### Suggested change (product call) Decide whether Maven remote **hits** are always proxy delivery, even if other formats redirect. --- ## Out of scope here Sidecar synthesis (Steps 12 / 16). The extra `.sha1` GET after a cold fill is expected until those land, even if we later emit checksums on the metadata response itself. Related to https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/286 Related to https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1745