## What this is **Last reconciled against `main` on 2026-09-14.** Rows record what was measured at filing time; the S08 bands have landed since, so re-check a row against `main` before acting on it. A single place to record every inconsistency the S08 remote conformance run found between the specs and the reference it validated against (a locally built Artifact Registry at `2fe35a9ce`). It is filed here rather than against Artifact Registry deliberately: for most of these it is not yet established whether the reference is wrong or the spec is, and several are deliberate reference behaviour that the spec simply does not describe. Splitting them into AR issues before that is decided would be filing bugs against choices. Full detail for each row is in the run's `validation/reference-quirks.md`, which carries the measurement, the evidence path and the reasoning. This table is the index and the classification. ## Classification - **Spec gap**: the spec does not say, or says something the protocol does not support. The fix is an amendment. - **Reference divergence**: the spec says, the reference does otherwise. The fix is one or the other, and the row says which is in question. - **Environment**: neither is wrong; a harness has to be configured a particular way, and a run that is not will mis-report. ## Reference divergences | # | Property | Spec says | Reference does | Status | | - | -------- | --------- | -------------- | ------ | | R16 | npm error body's `error` field | S06 §Error response shape: JSON with an `error` field **(string)** | `{"error":{"code":…,"message":…,"request_id":…}}`, an object. Consistent across all four cases measured (`404`, `405`, `409`, `403`) | **Filed against the reference.** Re-measured on staging 2026-09-14 with the npm CLI: `npm view` against Artifact Registry prints `[object Object]` where the same command against npmjs.org prints `Not found`. The registry's own `message` is correct and simply unreachable, because `npm-registry-fetch` resolves `body.error || body.message` and an object wins the chain. Filed as gitlab-org/ops/artifact-registry#1342. S06 is not amended: the arbiter here is the official client, not the spec, and the client cannot read the nested shape. Affects seven rows, not six: `npm.publish.name-validation` belongs with the two hosted rows named above. **Done. #1342 closed 2026-09-15T10:04Z and the change is live on staging**, measured 2026-09-15T12:05Z: `GET /ar-registry/npm/conformance-npm/no-such-package-xyz` returns `404 application/json; charset=utf-8` with `{"error":"package not found","code":"package_not_found","request_id":"be8d8b30-..."}`. `error` is a string. `npm.error.response-shape`, `npm.error.duplicate-publish-body` and `npm.publish.name-validation` all pass on run `1d383eea80ad4292`. The `new` row below is unchanged: the `401` still carries no envelope at all. | | new | npm `401` body | S06: "for **every** error response, the body is JSON with at least an `error` field" | Empty body, no `Content-Type`, only `WWW-Authenticate: Bearer realm="artifact-registry"` | Open. Distinct from R16: here there is no envelope at all rather than a differently-shaped one | | R10 | npm tarball path for a scoped name | S06 §Operation: Tarball download: the fetched URL is `dist.tarball` from the packument, "not a computed path" | Serves the tarball **only** at `@scope%2Fpkg`, `404`s the raw-slash form, and advertises the encoded form | Resolved in the suite: follow the advertisement, never compute. The reference is within its rights; the plan's computed path was the defect | | R8 | npm duplicate publish | S06 §Operation: Publish admits `409` **or** `403`. S08 §Downstream amendments assigns `403` to GitLab | `409 version_exists` | Spec-internal: S08's parenthetical is narrower than S06's rule and picks the branch the reference does not take | | R2 | OCI upstream bearer realm | S08 expects a remote OCI read to serve or `404` | `503`; the upstream client refuses a `WWW-Authenticate` bearer realm whose scheme is not `https` | **Not a defect.** The refusal is the security property the suite should be at least as strict about. Environment fixed with a TLS proxy | | R20 | OCI repository addressing | the suite builds `base + "/v2/" + image` with an image carrying no repository prefix, per the README's `--registry-url=.../remote-repo` | serves `/v2/` **at the root**, with the repository path as the image name's leading segments. `/conformance/container/oci-remote-tls/v2/` is a `404`; so is `/v2/conformance/<runID>/<slug>/tags/list` | **Resolved.** `--oci-name-prefix` (!268) and `--oci-upstream-name-prefix` (!277) teach the client a repository prefix, S07 gained §Configurable name prefix ahead of them, and #60's remaining half closed with !295. The proxy shim this run used is no longer needed. | ## Spec gaps | # | What the spec leaves open | Consequence | | - | ------------------------- | ----------- | | R1 / R4 / R5 | S08 pins that a cold-fill read differs from a warm read, but not how, and the transition is **four different behaviours**: Maven adds 4 headers; npm packument adds `ETag` only; OCI manifest adds nothing; OCI blob changes framing `chunked` to `Content-Length` | A cross-format assertion generalised from any one of them is wrong for the others The AC #23 half is now tracked as #78. | | R5 | `Accept-Ranges` on OCI blobs | Absent from every blob response, cold and warm, yet `Range` is honoured and returns `206` with `Content-Range`. A row asserting the advertisement would fail a working implementation Resolved in the suite: `oci.remote.blob-range` requires neither `Accept-Ranges` nor a declared `Content-Length`, and its catalog row cites this measurement. | | R7 | Duplicate-write verdict per format | Three different behaviours; Maven's depends on whether the bytes match (`200` idempotent vs `409 file_exists`) | | R13 | S08 AC #19's second clause (`none is under --upstream-url`) | Unreachable against this reference: its two bases are sibling paths, so the first clause always short-circuits. Separating them needs a nested upstream, which AR's URL scheme cannot express | | Open 4 | AC #19 permits a vacuous pass, and does not define "under" | "Every `dist.tarball` is under X" is trivially true over an empty `versions` map; a bare string prefix accepts a sibling repository | | Open 6 | AC #16 permits a vacuous pass | "Byte-identical to the first" is satisfied by two empty responses | | Open 5 | `tarball-relay` names `dist.integrity`, the hosted helper requires only one of shasum/integrity | Reused as-is the row would pass on a SHA-1 match against a relay that dropped integrity | Opens 1-7 were tracked in the plan's §Spec-amendment candidates (`docs/plans/2026-08-21-remote-conformance.md`). Opens 4, 5 and 6 have since been raised as #67's Amendments 4, 5 and 6. The OCI band's own candidates are #70 (Opens 5, 5a, 6-11) and the Maven band's are #76. ## Environment requirements | # | Requirement | What happens without it | | - | ----------- | ----------------------- | | R3 | `npm.public_registry_url` must be set | Every remote npm packument read is `500`, logged as `npm.public_registry_url is unset`. Also: whatever it is set to **is** `npm.remote.tarball-url-rewritten`'s expected host, so it is ground truth rather than config | | R9 | The suite must address the reference by a multi-label `.test` name, not a loopback literal | npm's client refuses a registry-advertised `dist.tarball` on a loopback or private literal, so every npm tarball row fails client-side before a request is issued | | R11 | For a proxy-side mutation on a redirected leg, the registry must **advertise** the proxy | The tarball leg follows the advertised `dist.tarball` straight past the proxy and the mutation silently does nothing. In the other direction, leaving the override in place fails `tarball-url-rewritten` against a correct registry | ## Suite-side findings, not about the reference Recorded here because they were found by the same run and two want their own MRs. | # | Finding | Status | | - | ------- | ------ | | R17 | `redact.ScanContent` does not redact a secret in a JSON field: its pattern needs the keyword followed directly by space/tab/colon/equals, and JSON's closing quote sits in between. `{"token":"secret"}` is not redacted; `token=secret` is. **54 production call sites, all three formats** | **Open, security.** Wants its own MR: over-redaction in a shared primitive fails invisibly, and the current `[^\r\n]*` would swallow every remaining JSON field Still live on `main`, and wider than this row records: `ScanContent` has no JSON handling at all, so `{"token":"s3cr3t"}`, `{"my token":"s3cr3t"}`, `{"password": "hunter2"}` and a nested `{"error":{...,"token":"s3cr3t"}}` are all returned verbatim, while `token=s3cr3t` and `authorization: Bearer abc` redact. Measured against `redact.go`'s `contentPattern` on 2026-09-14. Tracked as #61, whose Summary has been corrected to match its own §Until this lands. | | R18 | 52 nil-client constructions in `pkg/client/npm` tests share the process-wide `http.DefaultClient`, causing whole-repo flakes that pass in isolation 52 when measured; 82 `npm.New(` call sites across ten test files as of 2026-09-14, so the population grows with each new test file rather than staying put. | Open. One instance fixed as precedent; the rest land in one MR. Because the count grew rather than held, a guard against new instances is filed separately so the fix converges instead of being redone per test file. | | R12 | A non-text response body reached the operator's message on stderr, which applies no redaction | Fixed | | R14 | `mutproxy`'s own `extend-body` was broken (appended bytes without updating `Content-Length`, producing an empty response) and `strip-header` silently cannot remove the three headers net/http regenerates | Fixed; whole menu now asserted by `verify-mutations.sh` | | R19 | `pkg/client/oci` collapses an absent `Docker-Content-Digest` into a present-but-empty one: both response builders read through `resp.Header.Get`, which returns `""` for either. Probed on the wire, the information is there (`present=true, values=[""]`) and is discarded at the client seam | **Accepted, closed.** The collapse is now written down rather than latent: `docs/catalog/oci.md`'s `oci.remote.docker-content-digest` row states that an absent header and a present-but-empty one are one case at the client seam and that the row does not distinguish them. | ## The decision that was open R16, decided 2026-09-14: **S06 stands, the reference changes.** Filed as gitlab-org/ops/artifact-registry#1342. What settled it was not the spec. There is no authoritative npm rule here, and S06 says so: npm's own `restful-api-conventions.md` documents `{message: ...}` while `npm-registry-fetch` reads `body.error` first, and S06's rule is marked `Source: catalog-bound`. So "which document is right" had no answer. The official client did. `npm view` of a non-existent package against Artifact Registry prints `[object Object]`; the same command against npmjs.org prints `Not found`. Measured 2026-09-14 with npm 10.9.2. The argument recorded here earlier, that `npm-registry-fetch` builds its message as `body.error || body.message || JSON.stringify(body)` and an object short-circuits the chain, is what the measurement confirms. **Still undecided: the `new` row above.** npm's `401` carries no envelope at all, confirmed again 2026-09-14 (`content-length: 0`, no `Content-Type`, only `WWW-Authenticate`). #1342 does not cover it. ## How these were measured Every row above is a measurement against a live reference, not a reading of the code. The probes are kept and re-runnable: - `validation/ref-env/probe-error-shapes.sh`: the four npm error cases - `validation/ref-env/probe-disttags-agreement.sh`: the dist-tags surfaces - `validation/ref-env/verify-mutations.sh`: the harness's own mutation menu - `validation/ref-env/mutation-matrix-step{19,20,21,22,23,24}.sh`: the per-row negative controls One correction worth stating, because it nearly became a false accusation in this issue: `reference-quirks.md` originally recorded R8's body as `{"code":"version_exists",…}`, which is the *inner* object without its wrapper. Read from the ledger, that would have made R16 look like an internal inconsistency in the reference rather than a consistent design choice. Re-measuring all four error cases showed the reference is entirely consistent. The ledger is corrected. ## Staging run (2026-09-02): pre-run blockers Two rows found before any staging run, by reading the code rather than by measuring a response. Both block a track outright, so they are recorded here ahead of the run they would otherwise be found by. Numbered `S` to keep them distinct from the local-reference `R` rows above. | # | Classification | What | Consequence | | - | -------------- | ---- | ----------- | | S1 | **Spec gap** | S07 §Repository naming pins the OCI repository name to `conformance/{runID}/{slug-mapped}` (`repoNameForRunID`, `pkg/conformance/oci/env.go:315`), and the client builds every URL as `<--registry-url>/v2/<name>/...`. Artifact Registry routes `/v2/{slug}/container/{repository_name}/{image_name...}` and requires segment index 1 to be the literal `container` (`ContainerLiteralPos = 1`, enforced at `internal/format/oci/ociroute/ociroute.go:229`). The suite puts the run ID in that segment. | **OCI conformance cannot address an AR repository at all**, hosted or remote. Every request 404s at the router before it reaches a handler. `--registry-url` cannot carry the prefix: the `/v2/` literal is inserted after the base URL, not before the name. S07 never says what a target with a mandatory path prefix does, which is why this is a gap rather than a divergence. Unblocking needs either a configurable OCI name prefix (S07 amendment plus a new flag) or AR accepting a name without the `container` literal. | | S2 | **Suite-side** | `list --format <f> --repository-kind remote` returns the full hosted inventory for all three formats. No format module declares a `NeedsUpstream` descriptor (`grep -rn NeedsUpstream pkg/conformance/maven pkg/conformance/npm pkg/conformance/oci` is empty), and the catalog has no `## Remote` or `## Protocol baseline` section. | A remote run does not skip cleanly, it mis-runs. With `--upstream-url` the hosted rows execute against the remote repository and then `nothingEstablished` (`pkg/conformance/run.go:548`) exits `2` with `catalog has no seeding row`. With `--upstream-free-only` the same rows execute, the write rows fail because a remote refuses writes, and the run exits `1`. Neither exit code says anything about proxying, and neither is distinguishable from a real result without reading the diagnosis string. Gated on a format-specific child of #47. | | S3 | **Reference divergence** | R16's envelope shape, measured on staging rather than on the local reference: `{"error":{"code":"package_not_found",...}}`. Observed 2026-09-02 (curl), 2026-09-04 (the remote rows) and 2026-09-14 (npm CLI and curl, request ID `fa828c81-f3a9-4c7a-a089-707a6f17bfd1`). | Filed as gitlab-org/ops/artifact-registry#1342. Its four sibling findings from the 2026-09-02 run were filed as that project's #1126, #1127, #1129 and #1130; this one was annotated here instead, which is why it had no issue on their board for twelve days. **Fixed, see R16.** The nested shape has not gone away entirely: it still answers above the npm handler, which is S6 in the 2026-09-15 section. | | S4 | **Environment** | The edge in front of staging refuses a default-`User-Agent` HTTP client with `403`, `text/plain`, body `error code: 1010`, before the request reaches AR. curl's default passes; Python's `urllib` does not. | A probe that does not set a User-Agent records the edge's refusal as if it were the registry's. Related: that project's #1136 and #1132. | ### S1: how it was established ```shell # Suite-shaped name. Expect a router-level 404. curl -sSi -H "Authorization: Bearer $AR_TOKEN" \ "https://$AR_HOST/v2/conformance/probe123/oci-version-check/tags/list" # AR-shaped name, for contrast. curl -sSi -H "Authorization: Bearer $AR_TOKEN" \ "https://$AR_HOST/v2/$SLUG/container/conformance-oci/test-app/tags/list" ``` The wire measurement is pending a staging AR host; the route requirement itself is read off `ociroute.go:229`, which fails the request before any handler runs. ### S2: how to tell it is fixed ```shell diff <(regconf list --format maven --repository-kind hosted) \ <(regconf list --format maven --repository-kind remote) regconf list --format maven --repository-kind remote | grep '\.remote\.' grep -rn NeedsUpstream pkg/conformance/maven pkg/conformance/npm pkg/conformance/oci ``` An empty diff, no `.remote.` rows, and no `NeedsUpstream` hit is the current state on `main` at 7ca1a6b. ## Staging run (2026-09-15) Hosted Track B against `artifact-registry.staging.gitlab.com`, slug `ar-registry`, registry-conformance `9b7bd98b` (main tip, no patches). Maven 31/31, npm 29/37, OCI 36/47. Run IDs `2828c10999c8f4e8`, `1d383eea80ad4292`, `f40ec9f197c89961`. Full report: `2026-09-15-hosted-conformance-report.md`. Continuing the `S` numbering from the section above. | # | Classification | What | Consequence | | - | -------------- | ---- | ----------- | | S5 | **Reference divergence** | `Docker-Content-Digest` on a redirect-served blob `GET`. AR sets it on the `307`; the storage hop's `200` does not repeat it. `HEAD blob` → `200` with the header, `GET blob` → `307` with the header, then `200` without it. Measured 2026-09-15T12:00Z, run `f40ec9f197c89961`. | Seven rows fail on this one cause: `oci.blob.upload-monolithic`, `upload-single`, `upload-chunked`, `download`, `cross-repo-mount`, `oci.http.docker-content-digest`, `oci.concurrency.blob-uploads`. `oci.blob.head` passes, which localizes it: AR is not omitting the header, the storage hop is not carrying it. The suite side is settled and will not move: S07 §AR-style blob redirect delivery binds those rows to the final response, and the client reads the hop's value only to name it in the failure message. The remaining decision is the reference's. Annotated on that project's #434, whose question 1 asks the mirror image of this. | | S6 | **Reference divergence** | Two error envelope shapes on one host. The npm handler returns `{"error":"package not found","code":"package_not_found","request_id":...}`; the router returns `{"error":{"code":"not_found","message":"not found","request_id":...}}` for `https://<host>/-/ping` and `https://<host>/<slug>/-/ping`. Measured 2026-09-15T12:15Z. | No conformance row addresses the router paths, so nothing failed on it. Recorded because a client that parses one shape meets the other on the same host, and because it is the shape R16 had before #1342 fixed the npm handler. | | S7 | **Reference divergence** | `GET /api/v1/<slug>/repositories` returns 20 entries against a `repositories_count` of 43, ignores `page` and `per_page`, and carries no cursor, no total and no pagination block. The list is alphabetical and stops at `conformance-maven-upstream`. Measured 2026-09-15T11:50Z. | A caller with more than 20 repositories cannot enumerate them, and nothing in the response says the list is partial. Confirming `conformance-npm` and `conformance-oci` existed needed `GET /api/v1/<slug>/repositories/<name>` per name. Management API, not a protocol surface, so no conformance row covers it. | | S8 | **Spec gap** | `npm.audit.bulk-advisories` drives `npm audit --package-lock-only`, and the CLI POSTs `/-/npm/v1/security/audits/quick`, not the `/-/npm/v1/security/advisories/bulk` route the row's framing assumes. Measured 2026-09-15T12:00Z from the CLI's own stderr. | The row cannot pass by serving the bulk route alone, so a red result here is not evidence that route is missing. Whether the row should drive the CLI at all, given the CLI chooses the endpoint, is S06's to settle. Annotated on that project's #1130. | | S9 | **Spec gap** | `dist-tags.latest` reconciliation. S06 §Operation: Publish requires it to be semver-aware and states "npmjs.org implements this", sourced only to our own catalog row. That project's #1125 was closed working-as-intended after a product call, citing npmjs as passive server-side with the guard in npm CLI 11 ([npm/cli!7939](https://github.com/npm/cli/pull/7939), a breaking change). The two claims about npmjs.org's server behaviour contradict each other. Measured 2026-09-15T12:20Z: three versions published ascending then descending, `dist-tags={"latest":"1.5.0"}` against `versions=["1.0.0","1.5.0","2.0.0"]`. | `npm.publish.latest-tag-semver` (`critical`, `done`) rests on the S06 claim, and marks both Artifact Registry and GitLab Package Registry non-conformant. The row is HTTP-only, so it bypasses any CLI-side guard, meaning npmjs.org itself would fail it. Settle the claim before calling the reference non-conformant. Filed as #86. |