## Proposal **👉🏻 Project URL:** https://gitlab.com/gitlab-org/ops/registry-conformance Build a standalone conformance test tool that validates any Artifact Registry implementation against the supported protocols (Maven, npm, Container/OCI) from outside the service boundary. The tool uses real native clients and direct HTTP calls to exercise the full protocol surface, producing a structured pass/fail report per capability. This tool serves two purposes: 1. **CI gate**: Run on every push to catch protocol regressions immediately, regardless of who wrote the code. 2. **Agent safety net**: AI agents implementing features validate their work against the conformance suite without knowledge of service internals. If conformance passes, the implementation is correct. Living outside the Artifact Registry codebase is a deliberate choice. Agents working on the service cannot modify the conformance tests to bypass failures caused by bugs. The tests are an immutable contract that the implementation must satisfy. ## Motivation The [Artifact Registry PoC](https://gitlab.com/jdrpereira/artifact-registry-poc) relied on [7 e2e shell scripts](https://gitlab.com/jdrpereira/artifact-registry-poc/-/tree/main/scripts) to validate the implementation. These were minimal scripts built to unblock the PoC, yet they caught integration issues that unit tests missed: auth token exchange failures, scope format mismatches, pre-computed counter bugs, and metadata format differences between clients. They also enabled AI agents to self-validate, running the suite after each change and confirming correctness without human intervention. The production implementation requires a proper conformance test tool. The OCI ecosystem has an [official conformance suite](https://github.com/opencontainers/distribution-spec/tree/main/conformance) that validates any registry against the Distribution Spec. No equivalent exists for Maven or npm: **npm**: The closest tools are [`abstract-npm-registry`](https://github.com/godaddy/abstract-npm-registry) (GoDaddy) and [`registry-validator`](https://github.com/strongloop-community/registry-validator) (StrongLoop). Both are unmaintained (last updated \~2014) and target the CouchDB-era registry API. They predate scoped packages, abbreviated packuments, and provenance. Not usable as-is. **Maven**: No protocol-level conformance tooling exists at all. Available tools like [`repository-validator`](https://github.com/jdcasey/repository-validator) and Debian's `mh_checkrepo` check static file consistency after the fact, not HTTP protocol behavior. Maven Core Integration Tests test Maven itself, not repository managers. Neither format has usable tooling. We must build our own. ## Design ### Structure A single tool with three format modules (more to come), following the OCI conformance suite model: - Standalone binary that depends only on the public API - Runs locally (against GDK or a standalone service) and in GitLab CI with identical behavior - Configurable target URL, credentials, format selection and testing scopes (all enabled by default) - Structured test report with pass/fail per capability - Exit code reflects overall pass/fail for CI integration ### Test modes The tool operates in two independent modes, each producing its own report: **Protocol mode**: Tests format-specific protocol correctness with auth disabled on the target service. Agents run this mode continuously while building features. It answers: "does the protocol work?" **Auth mode**: Tests authentication and authorization behavior. Requires a JWT issuer: either a real GitLab Rails instance (via GDK) or a built-in mock issuer that ships with the tool. The mock issuer mints JWTs signed with a known key and exposes a JWKS endpoint. Because the Go service verifies tokens against JWKS regardless of who issued them, the mock works without Rails. It tests token exchange, scope enforcement, permission boundaries (read-only tokens rejected on push, wrong org rejected, expired tokens return 401), and visibility rules. The mock issuer generates edge cases (expired tokens, wrong scopes, missing claims) more easily than a real Rails instance. Both modes run independently or together. Protocol mode requires only the target service. Auth mode requires GDK or the built-in mock issuer. ### Test cases The test cases below are an initial proposal based on protocol research and PoC experience. They need to be validated for completeness before implementation. #### Maven Primary driver: `mvn` CLI. Supplemented with direct HTTP calls for edge cases the CLI abstracts away. ##### Local <details> <summary>Click to expand</summary> | \# | Test case | Driver | Priority | Status | Notes | |----|-----------|--------|----------|--------|-------| | 1 | Release upload (JAR, POM, checksums, metadata) | `mvn` | critical | not started | `mvn deploy` uploads JAR, POM, checksums, and triggers metadata update | | 2 | Release download (`dependency:resolve`) | `mvn` | critical | not started | `mvn dependency:resolve` fetches a previously deployed artifact | | 3 | SNAPSHOT upload with timestamped filenames | `mvn` | critical | not started | Build number must increment atomically | | 4 | SNAPSHOT download via metadata-driven filename | `mvn` | critical | not started | Client reads `maven-metadata.xml`, constructs timestamped filename | | 5 | Concurrent SNAPSHOT deploys don't corrupt files | `mvn`+parallel | critical | not started | Two parallel deploys of the same SNAPSHOT must not corrupt stored files or metadata. Race condition fixed via unique index (!196170) | | 6 | `maven-metadata.xml` correctness after multiple version uploads | HTTP | critical | not started | `<latest>`, `<release>`, `<versions>` list must be correct | | 7 | `<release>` correctness after out-of-order deploys | HTTP | critical | not started | Deploy 1.0.0 then 0.9.0: `<release>` must remain 1.0.0. `<release>` excludes SNAPSHOTs | | 8 | HEAD request support | HTTP | critical | not started | Maven probes artifact existence via HEAD before every download | | 9 | `Expect: 100-continue` on PUT | HTTP | high | not started | Maven wagon-http sends this on every PUT; handled by web server layer (Workhorse/nginx) | | 10 | Classifier support (sources, javadoc, tests) | `mvn` | high | not started | `{artifactId}-{version}-{classifier}.jar` | | 11 | SNAPSHOT + classifier combined | `mvn` | high | not started | Timestamped name with classifier; `<snapshotVersions>` lists each classifier | | 12 | Checksum verification (SHA-1, MD5) | HTTP | high | not started | `.sha1` and `.md5` files contain hex-encoded hash only | | 13 | Conditional GET (`If-Modified-Since`, `ETag`, `304 Not Modified`) | HTTP | high | optional | GitLab doesn't support this — will fail against GDK | | 14 | Parent POM recursive resolution | `mvn` | high | not started | `<parent>` in POM triggers additional GETs | | 15 | Path layout compliance (`groupId` as directory structure) | HTTP | high | not started | Dots become slashes in groupId only | | 16 | Metadata consistency after async sync | HTTP | high | not started | After deploy + metadata sync completes, `maven-metadata.xml` must reflect the new version. GitLab syncs async via Sidekiq. | | 17 | Content-Type correctness on responses | HTTP | high | not started | `text/plain` for checksums, `application/octet-stream` for JARs | | 18 | SNAPSHOT version-level metadata preservation | HTTP | high | not started | Version-level `maven-metadata.xml` uploaded by client must be retrievable as-is (GitLab stores client-uploaded metadata, doesn't generate it server-side) | | 19 | Duplicate release rejection (when configured) | HTTP | high | not started | Second PUT of same release GAV → 4xx when duplicates disallowed. GitLab allows duplicates by default; test requires `maven_duplicates_allowed=false` | | 20 | SHA-1 upload verification (sidecar match) | HTTP | high | not started | Upload `.sha1` for existing file → 204 on match, 409 on mismatch | | 21 | `.sha256` and `.sha512` checksum files | HTTP | medium | not started | Maven 3.9+ uploads these. GitLab generates them for server-created metadata; for client uploads, only present if client uploads them. | | 22 | Deep and single-segment groupId paths | HTTP | medium | not started | `com.a.b.c.d.e` and `mygroup` | | 23 | POM-only deployment (`<packaging>pom</packaging>`) | `mvn` | medium | not started | No JAR uploaded | | 24 | Classifier-only download | `mvn` | medium | not started | `mvn dependency:resolve -Dclassifier=sources` | | 25 | `Transfer-Encoding: chunked` on PUT | HTTP | medium | optional | Depends on Workhorse/nginx layer | | 26 | Upload ordering resilience (partial deploy state) | HTTP | medium | not started | POM exists before JAR; GET JAR → 404 not 500 | | 27 | `<lastUpdated>` timestamp format | HTTP | medium | not started | Must be `yyyyMMddHHmmss` (14 digits, no dot) | | 28 | Non-unique SNAPSHOT fallback | HTTP | low | optional | Legacy Maven 2.x behavior. Unclear if GitLab supports resolving `artifact-SNAPSHOT.jar` to latest timestamped version. | | 29 | Retry behavior on 5xx errors | HTTP | low | optional | Not practically testable against healthy GDK without fault injection | | 30 | Empty/zero-byte file handling | HTTP | low | not started | Some artifacts are legitimately empty | | 31 | PGP signature files (.asc) | HTTP | low | not started | Stored like any sidecar file | | 32 | Maven plugin metadata (`<plugins>` XML) | HTTP | low | optional | groupId-level metadata for plugin prefix resolution (Maven metadata spec) | </details> ##### Errors <details> <summary>Click to expand</summary> | \# | Test case | Driver | Priority | Status | Notes | |----|-----------|--------|----------|--------|-------| | 1 | 404 for missing artifact | HTTP | high | not started | GET nonexistent GAV returns 404; response body must be parseable (not an HTML error page) | | 2 | RFC 9457 problem details format | HTTP | high | not started | Error responses must include `Content-Type: application/problem+json` with `type`, `title`, and `status` fields. `mvn` and `gradle` read the `detail` field and surface it in build output — significant UX improvement over opaque HTTP errors. GitLab Package Registry does not implement RFC 9457 (tracked in #595487); this test will fail against GitLab. | | 3 | 401/403 on unauthorized access | HTTP | high | not started | Requests without valid credentials or with insufficient permissions return 401/403 with a machine-readable error body | | 4 | Duplicate release error body | HTTP | medium | not started | Second PUT of same release when duplicates are disallowed (see local `#19`) returns a 4xx with an informative, machine-readable error body | </details> ##### Virtual <details> <summary>Click to expand</summary> | \# | Test case | Driver | Priority | Status | Notes | |----|-----------|--------|----------|--------|-------| | 1 | Upstream resolution order | `mvn` | critical | not started | First upstream with the artifact wins | | 2 | Metadata merging across upstreams | HTTP | critical | not started | `maven-metadata.xml` must merge versions from all upstreams | | 3 | Cache behavior and TTL | HTTP | high | not started | | | 4 | Health-degraded upstream | HTTP | high | not started | Healthy upstreams still return results when one upstream is down | | 5 | Allow/deny rule filtering | HTTP | high | not started | Rules block/allow specific coordinates | </details> #### npm Primary driver: `npm` CLI. Supplemented with direct HTTP calls for packument format validation and edge cases the CLI handles silently. ##### Local <details> <summary>Click to expand</summary> | \# | Test case | Driver | Priority | Status | Notes | |----|-----------|--------|----------|--------|-------| | 1 | Publish package (`npm publish`) | `npm` | critical | not started | PUT `/{name}` (unscoped) or `/@scope%2Fname` (scoped) with `_attachments` containing base64-encoded tarball. Test both name formats — scopes are a naming convention, not a routing mechanism. | | 2 | `latest` tag auto-assignment and semver-aware update | HTTP | critical | not started | First publish → `latest` points to it. Publish higher semver → `latest` updates. Publish lower semver → `latest` must NOT update. | | 3 | Install by exact version | `npm` | critical | not started | `npm install pkg@1.0.0`; fetches packument, then tarball | | 4 | Install by semver range | `npm` | critical | not started | `^1.0.0` must resolve to highest matching version. Pre-release versions (e.g. `2.0.0-beta.1`) must be excluded from range matching. Recommendation: Change priority to High. Reasoning: semver resolution happens client-side from the packument — the registry's job is just to return correct version data, which is already tested by #7 and #21. | | 5 | Install by dist-tag | `npm` | critical | not started | `npm install pkg@beta`; resolves `dist-tags.beta` to a version. Recommendation: Change priority to High. Reasoning: depends on dist-tags working (#14– #16). Important but secondary to version-pinned installs — lockfile-based installs (the most common CI path) use exact versions. | | 6 | Abbreviated packument (`Accept: application/vnd.npm.install-v1+json`) | HTTP | critical | not started | Response must contain only abbreviated fields: `name`, `modified`, `dist-tags`, `versions` (with `name`, `version`, `dist`, dependencies). Must NOT include `_id`, `_rev`, `time`, `readme`. GitLab always returns abbreviated-like metadata regardless of Accept header (`generate_metadata_service.rb` — no Accept header inspection). Note: this test would trivially pass against GitLab because it never includes full fields, not because it correctly differentiates formats. Recommendation: Change priority to Medium. Reasoning: returning a full packument instead of abbreviated won't break installs — npm handles both. It's a performance optimization (less data over the wire), not a correctness issue. GitLab doesn't differentiate formats today. | | 7 | Full packument (no `Accept` header or `application/json`) | HTTP | critical | not started | Must include `_id`, `_rev`, `time`, `readme`, hoisted fields from latest version (`description`, `author`, `license`, etc.). GitLab Package Registry returns none of these — metadata response contains only `name`, `versions`, and `dist-tags` (`generate_metadata_service.rb:47-51`). Will fail against GitLab. | | 8 | SRI integrity field format (`sha512`) | HTTP | critical | not started | `dist.integrity` must be `sha512-{base64}`; `sha256-` prefix causes npm `EINTEGRITY` errors. GitLab Package Registry does not return `integrity` — only `shasum` (SHA-1) in `dist` object (`generate_metadata_service.rb:114-117`). Installs still work via `shasum` fallback. Recommendation: Change priority to High. Reasoning: GitLab has operated without `integrity` for years — installs work via `shasum` fallback. Important for a modern registry but not a blocker. | | 9 | `shasum` field in dist object | HTTP | critical | not started | `dist.shasum` is SHA-1 hex string; required by older npm clients as legacy fallback. GitLab Package Registry returns only `shasum` (no `integrity`), so this is the sole verification path today. Recommendation: Change priority to High. Reasoning: legacy fallback field. Modern npm uses `integrity` first. GitLab relies on `shasum` exclusively, proving it's sufficient but not the primary path. | | 10 | Tarball integrity verification on download | HTTP | critical | not started | Downloaded tarball SHA-1 must match `dist.shasum` from packument. If `dist.integrity` is present, SHA-512 must also match. GitLab Package Registry only provides `shasum` (no `integrity`), so only SHA-1 verification is possible against GitLab. | | 11 | Tarball download and URL correctness | HTTP | high | not started | Unscoped: GET `/{name}/-/{name}-{version}.tgz`. Scoped: GET `/@scope/{name}/-/{name}-{version}.tgz` (filename drops scope prefix). Packument `dist.tarball` must be usable by npm client without modification. Recommendation: Change priority to Critical. Reasoning: tarball download is the final step of every `npm install` — if the URL is wrong or download fails, nothing installs. | | 12 | Dist-tag add (`npm dist-tag add`) | `npm` | high | not started | PUT `/-/package/{name}/dist-tags/{tag}` | | 13 | Dist-tag remove (`npm dist-tag rm`) | `npm` | high | not started | DELETE `/-/package/{name}/dist-tags/{tag}` | | 14 | Dist-tag list (`npm dist-tag ls`) | `npm` | high | not started | GET `/-/package/{name}/dist-tags` | | 15 | Unpublish specific version | `npm` | high | optional | GET packument with `?write=true`, PUT modified packument without the version, DELETE tarball. Requires `_rev`. GitLab Package Registry does not support npm unpublish — no DELETE route for packages exists (`npm_endpoints.rb` only has DELETE for dist-tags; `npm unpublish` absent from [supported commands](https://docs.gitlab.com/user/packages/npm_registry/)). Recommendation: Change priority to Low. Reasoning: GitLab does not support npm unpublish. AR likely handles deletion via its management API, not the npm protocol. | | 16 | Unpublish entire package | `npm` | high | optional | DELETE `/{name}/-rev/{_rev}`. GitLab Package Registry does not support this — no package-level DELETE route exists (same as #15; `npm unpublish` absent from [supported commands](https://docs.gitlab.com/user/packages/npm_registry/)). Recommendation: Change priority to Low. Reasoning: same as #15 — GitLab does not support npm unpublish. Deletion via management API. | | 17 | Deprecate version or range (`npm deprecate`) | `npm` | high | not started | PUT packument without `_attachments`; `deprecated` field set on matching versions | | 18 | `_rev` field in packument responses | HTTP | high | not started | CouchDB-era concurrency token. Required by standard npm protocol for unpublish and deprecate. GitLab Package Registry does not return `_rev` — it implements `npm deprecate` via a custom `Npm-Command: deprecate` header instead (`npm_project_packages.rb:50`). Recommendation: Change priority to Low. Reasoning: CouchDB artifact. GitLab doesn't return `_rev` and still supports deprecate via `Npm-Command` header. The only consumers are unpublish (not supported) and deprecate (works without it). | | 19 | Custom tag on publish (`npm publish --tag beta`) | `npm` | high | not started | Published version gets `beta` tag; `latest` must NOT be updated | | 20 | Content-Type correctness on responses | HTTP | high | not started | `application/json` for packuments, `application/octet-stream` for tarballs | | 21 | Error response format (npm-compatible JSON) | HTTP | high | not started | Error responses must be JSON with `error` field (e.g. `{"error": "not_found"}`). npm client parses this for user-facing messages. | | 22 | `npm audit` (`POST /-/npm/v1/security/advisories/bulk`) | `npm` | high | not started | Registry must either proxy the request to a vulnerability source (307 redirect) or return an empty advisory array `[]`. GitLab supports this — forwards to npmjs.org if package forwarding is enabled, returns `[]` otherwise (`npm_endpoints.rb`). `npm audit` is a [supported command](https://docs.gitlab.com/user/packages/npm_registry/). | | 23 | Tarball download follows redirects (object storage) | HTTP | high | not started | Registry may serve tarballs via 302 redirect to a signed object storage URL instead of streaming the file directly. npm client must follow redirects. GitLab does this when S3 direct download is enabled (`present_package_file!` in `npm_project_packages.rb`). | | 24 | Un-deprecate (set `deprecated` to `""`) | `npm` | medium | not started | Empty string removes deprecation message. Recommendation: Change priority to High. Reasoning: pairs with deprecate (#17) — same endpoint, different payload. If AR supports deprecate, it should support un-deprecate. GitLab fully supports this. | | 25 | Duplicate version rejection | HTTP | medium | not started | Second publish of same name@version → error. Status code is registry-specific (npmjs.com: 409 `EPUBLISHCONFLICT`, GitLab: 403). Recommendation: Change priority to Critical. Reasoning: package immutability is a supply chain security fundamental — if the same version is published twice with different content, downstream consumers get inconsistent artifacts. | | 26 | Scoped package URL encoding edge cases | HTTP | medium | not started | `@scope%2Fname` (single-encoded) must work. Double-encoding (`%40scope%2Fname`) behavior should be defined. | | 27 | Package name validation (214 char limit, special chars) | HTTP | medium | not started | Names must follow npm naming rules: ≤214 chars, no uppercase, no leading dots/underscores in unscoped | | 28 | Concurrent publishes of same version | `npm`+parallel | medium | not started | Must serialize or reject; no partial/corrupt state. Recommendation: Change priority to Critical. Reasoning: as a GitLab product, AR will be heavily used in CI — concurrent publishes are a when-not-if scenario, and silent data corruption is the worst possible outcome. | | 29 | Large packument performance (50+ versions) | HTTP | medium | not started | Packument with many versions must remain performant and correctly structured | | 30 | `npm whoami` (`GET /-/whoami`) | `npm` | medium | not started | Returns `{"username": "..."}`. Baseline token validity check. GitLab Package Registry does not support this — no `/-/whoami` route exists. Recommendation: Change priority to Low. Reasoning: GitLab doesn't support this. AR handles auth validation through its own mechanisms. | | 31 | `npm ping` (`GET /-/ping`) | HTTP | medium | not started | Returns `{}`. Trivial health check. GitLab Package Registry does not support this — no `/-/ping` route exists. Recommendation: Change priority to Low. Reasoning: trivial endpoint rarely used in practice. GitLab doesn't support it. AR has its own health endpoints via management API. | | 32 | `npm view` for specific version (`GET /{name}/{version}`) | `npm` | medium | not started | Returns single version object, not full packument. GitLab Package Registry does not support this — no version-specific route exists (`npm_endpoints.rb` only has `GET /{name}` for full packument). | | 33 | `time` field in full packument | HTTP | medium | not started | Object with per-version ISO 8601 timestamps plus `created` and `modified` keys. GitLab Package Registry does not include this — no `time` key in metadata response (`generate_metadata_service.rb:47-51`). Recommendation: Change priority to Low. Reasoning: not required for install/publish. GitLab doesn't include it. AR's management API likely exposes timestamps separately. | | 34 | Empty `dist-tags` on publish (auto-create `latest`) | HTTP | low | not started | If publish payload omits dist-tags, registry must create `latest` pointing to published version | | 35 | Multiple versions in single publish payload | HTTP | low | optional | Technically valid per CouchDB-era API; unclear if modern registries support it | | 36 | `dist.fileCount` and `dist.unpackedSize` in packument | HTTP | low | not started | Present in npm packuments since Feb 2018. `npm view` displays these. GitLab Package Registry does not return them — `dist` object contains only `shasum` and `tarball` (`generate_metadata_service.rb:114-117`). | | 37 | Conditional GET on packument (`ETag` / `If-None-Match` / `304`) | HTTP | low | optional | npm client caches packuments and sends conditional requests. GitLab does not support this. Without it, every install downloads the full packument — functional but inefficient. Optimization opportunity for later (promote to High when AR targets production performance). | | 38 | `npm search` (`GET /-/v1/search`) | HTTP | low | optional | Full-text search endpoint. GitLab Package Registry does not support this. Users discover packages through the registry UI, not `npm search`. | </details> ##### Errors <details> <summary>Click to expand</summary> | \# | Test case | Driver | Priority | Status | Notes | |----|-----------|--------|----------|--------|-------| | 1 | 404 for non-existent package | HTTP | high | not started | GET /package for a package that does not exist returns 404 with a JSON body containing an "error" field (not an HTML error page). GitLab Package Registry returns {"message": "404 Package not found", "error": "Package not found"} | | 2 | 401/403 on unauthorized access | HTTP | high | not started | Requests without valid credentials or with insufficient permissions return 401/403 with a machine-readable JSON error body | | 3 | Duplicate publish error body | npm | medium | not started | npm publish of an already-published name@version returns an error with a JSON body. GitLab Package Registry returns 403; consider 409 Conflict instead (matches npmjs.org behavior and is semantically more accurate — 403 implies an authorization problem, not a version conflict) | | 4 | JSON error format | HTTP | high | not started | Error responses must return Content-Type: application/json with at least an "error" field per npm registry conventions (https://github.com/npm/registry/blob/main/docs/restful-api-conventions.md). GitLab Package Registry uses {"message": "...", "error": "..."} where "message" redundantly includes the status code; consider dropping the status-code prefix for cleaner output | </details> ##### Virtual <details> <summary>Click to expand</summary> | \# | Test case | Driver | Priority | Status | Notes | |----|-----------|--------|----------|--------|-------| | 1 | Upstream resolution order | `npm` | critical | not started | First upstream with the package wins | | 2 | Packument merging across upstreams | HTTP | critical | not started | `versions` and `dist-tags` from multiple upstreams must merge correctly | | 3 | Tarball URL rewriting | HTTP | critical | not started | Proxied tarball URLs in merged packument must point to virtual repo endpoint, not upstream | | 4 | Cache behavior and TTL | HTTP | high | not started | Cached packument served until TTL expires | | 5 | Cache invalidation on upstream update | HTTP | high | not started | Stale packument refreshed when upstream has new versions | | 6 | Upstream failure handling | HTTP | high | not started | Healthy upstreams still return results when one upstream is down | | 7 | Metadata forwarding when package not found locally | HTTP | high | not started | When a package doesn't exist in the local registry and forwarding is enabled, registry should redirect (302) to the upstream (e.g. npmjs.org). GitLab supports this (`npm_project_packages.rb` — redirects to `registry.npmjs.org` when packages empty and forwarding enabled). | </details> #### Container/OCI Leverage the existing [OCI conformance suite](https://github.com/opencontainers/distribution-spec/tree/main/conformance). See the [conformance README](https://github.com/opencontainers/distribution-spec/blob/main/conformance/README.md) for the full list of covered workflows. #### Local <details> <summary>Click to expand</summary> | \# | Test case | Driver | Priority | Status | Notes | |----|-----------|--------|----------|--------|-------| | 1 | `/v2/` version check endpoint | HTTP | critical | not started | `GET /v2/` MUST return `200 OK` if the registry implements this specification ([end-1](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 2 | Blob upload (monolithic POST+PUT) | HTTP | critical | not started | `POST /v2/<name>/blobs/uploads/` MUST return `202 Accepted` with `Location` header ([end-4a](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). `PUT <location>?digest=<digest>` MUST return `201 Created` with `Location` pointing to pullable blob URL ([end-6](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 3 | Blob upload (single POST) | HTTP | critical | not started | `POST /v2/<name>/blobs/uploads/?digest=<digest>` with blob body. MUST return `201 Created` if supported; registries that do not support it SHOULD return `202 Accepted` with `Location` for fallback to POST+PUT ([end-4b](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 4 | Blob download | HTTP | critical | not started | `GET /v2/<name>/blobs/<digest>` MUST return `200 OK` with blob content. MUST include `Docker-Content-Digest` header matching the digest ([end-2](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Clients SHOULD verify response body matches requested digest. Covered by upstream OCI conformance suite (v1.0.1). | | 5 | Blob existence check (HEAD) | HTTP | critical | not started | `HEAD /v2/<name>/blobs/<digest>` MUST return `200 OK` with `Docker-Content-Digest` and `Content-Length` headers ([end-2](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 6 | Manifest push (by tag) | `crane` | critical | not started | `PUT /v2/<name>/manifests/<tag>` with OCI manifest. MUST return `201 Created` with `Location` header. `Docker-Content-Digest` MUST equal client-provided digest ([end-7](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 7 | Manifest push (by digest) | HTTP | critical | not started | `PUT /v2/<name>/manifests/<digest>` with manifest body. MUST return `201 Created`. Digest in URL must match manifest content ([end-7](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Not covered by upstream OCI conformance suite. | | 8 | Manifest pull (by tag) | `crane` | critical | not started | `GET /v2/<name>/manifests/<tag>` MUST return `200 OK` with manifest body. `Content-Type` SHOULD match what was pushed. MUST include `Docker-Content-Digest` header ([end-3](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 9 | Manifest pull (by digest) | HTTP | critical | not started | `GET /v2/<name>/manifests/<digest>` MUST return `200 OK`. Client SHOULD verify returned content matches requested digest ([end-3](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 10 | Manifest existence check (HEAD) | HTTP | critical | not started | `HEAD /v2/<name>/manifests/<reference>` MUST return `200 OK` with `Docker-Content-Digest` and `Content-Length` ([end-3](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 11 | Tag listing | HTTP | critical | not started | `GET /v2/<name>/tags/list` MUST return `200 OK` with JSON body `{"name":"<name>","tags":[...]}`. Tags MUST be in lexical order ([end-8a](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 12 | ~~OCI conformance suite (full)~~ | ~~OCI binary~~ | ~~critical~~ | skipped, see https://gitlab.com/gitlab-org/gitlab/-/work_items/591953#note_3253335084 | ~~Run upstream OCI conformance suite (`conformance.test` binary) with Pull, Push, Content Discovery, and Content Management workflows enabled. Produces `junit.xml` and `report.html`. Validated locally: 59 passed, 0 failed, 3 skipped against Container Registry (v1.0.1). v1.1/latest not yet passing (referrers API not implemented).~~ | | 13 | Tag listing with pagination (`n` and `last`) | HTTP | high | not started | `GET /v2/<name>/tags/list?n=<int>&last=<tagname>` MUST return up to `<int>` tags after `<tagname>`. MAY include `Link` header with `rel="next"` per RFC 5988. When `n=0`, MUST return empty list and MUST NOT include `Link` header ([end-8b](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 14 | Chunked blob upload (PATCH) | HTTP | high | not started | `POST` to get session, then `PATCH <location>` with `Content-Range` for each chunk, then `PUT` to close. Each `PATCH` MUST return `202 Accepted` with `Location` and `Range` headers ([end-5](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 15 | Chunked upload out-of-order rejection | HTTP | high | not started | Upload chunk with wrong `Content-Range` offset. Registry MUST respond with `416 Requested Range Not Satisfiable` ([end-5](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 16 | Cross-repository blob mount | HTTP | high | not started | `POST /v2/<name>/blobs/uploads/?mount=<digest>&from=<other_name>` MUST return `201 Created` if blob exists in source repo. Registry MAY return `202 Accepted` if mount not supported ([end-11](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 17 | Manifest delete (by digest) | HTTP | high | not started | `DELETE /v2/<name>/manifests/<digest>` MUST return `202 Accepted`. Subsequent `GET` by digest and any tag pointing to that digest MUST return `404` ([end-9](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 18 | Tag delete | HTTP | high | not started | `DELETE /v2/<name>/manifests/<tag>` MUST return `202 Accepted`. If tag deletion is disabled, MUST respond with `400 Bad Request` or `405 Method Not Allowed` ([end-9](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 19 | Blob delete | HTTP | high | not started | `DELETE /v2/<name>/blobs/<digest>` MUST return `202 Accepted`. If not found, MUST return `404`. If disabled, MUST respond with `400` or `405` ([end-10](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 20 | Manifest with `subject` field (referrers) | HTTP | high | not started | `PUT` manifest with `subject` referencing another manifest. Registry that supports the referrers API MUST respond with `OCI-Subject: <subject digest>` header. Not covered by upstream OCI conformance suite (v1.1 feature). | | 21 | Manifest with `subject` referencing non-existent manifest | HTTP | high | not started | Registry MUST accept a valid manifest with `subject` referencing a manifest that does not exist in the repository. Push-order independence. Not covered by upstream OCI conformance suite (v1.1 feature). | | 22 | Manifest blob reference validation | HTTP | high | not started | Registry MAY reject manifest with descriptors referencing manifests or blobs that do not exist. When rejected, MUST result in one or more `MANIFEST_BLOB_UNKNOWN` errors ([code-5](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#error-codes)). Registry MUST accept manifest with `subject` referencing non-existent manifest. Not covered by upstream OCI conformance suite. | | 23 | Referrers API listing | HTTP | high | not started | `GET /v2/<name>/referrers/<digest>` MUST return `200 OK` with `Content-Type: application/vnd.oci.image.index.v1+json`. MUST NOT return `404`. Empty `manifests` array if no referrers ([end-12a](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Not covered by upstream OCI conformance suite (v1.1 feature). | | 24 | Referrers API filtering by `artifactType` | HTTP | medium | not started | `GET /v2/<name>/referrers/<digest>?artifactType=<type>` SHOULD be supported. If filtering is applied, response MUST include `OCI-Filters-Applied: artifactType` header. Multiple filters → comma-separated list ([end-12b](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Not covered by upstream OCI conformance suite (v1.1 feature). | | 25 | Referrers tag schema fallback | HTTP | medium | not started | When referrers API returns `404`, client MUST fallback to pulling tag per [referrers tag schema](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#referrers-tag-schema) (e.g. `sha256-<hex>`). Not covered by upstream OCI conformance suite (v1.1 feature). | | 26 | Content negotiation via `Accept` header | HTTP | high | not started | Client SHOULD include `Accept` header. `Content-Type` MUST match what was pushed. Registry SHOULD NOT include parameters on `Content-Type` ([end-3](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints), [Pulling Manifests](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pulling-manifests)). Not covered by upstream OCI conformance suite. | | 27 | `Docker-Content-Digest` header correctness | HTTP | high | not started | MUST be present on successful manifest and blob GET/HEAD responses. Value MUST match digest of returned body if used by client ([end-2](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints), [end-3](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Not covered by upstream OCI conformance suite as a dedicated validation. | | 28 | Content-Type correctness on responses | HTTP | high | not started | `Content-Type` SHOULD match what was pushed as the manifest's `Content-Type`. Registry SHOULD NOT include parameters on `Content-Type` ([Pulling Manifests](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pulling-manifests)). Not covered by upstream OCI conformance suite. | | 29 | Manifest payload size limit | HTTP | medium | not started | Registry SHOULD enforce a max manifest size. SHOULD respond with `413 Payload Too Large` when exceeded. Client and registry SHOULD support at least 4 MB ([end-7](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Not covered by upstream OCI conformance suite. | | 30 | Blob Range request support | HTTP | medium | not started | Registry SHOULD support `Range` request header per [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-range-requests) ([Pulling Blobs](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pulling-blobs)). Not covered by upstream OCI conformance suite. | | 31 | Upload session status check | HTTP | medium | not started | `GET <location>` on an active upload MUST return `204 No Content` with `Location` and `Range` headers showing current progress ([end-13](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Not covered by upstream OCI conformance suite. | | 32 | Upload session cancellation | HTTP | medium | not started | `DELETE <location>` during a blob upload. Response SHOULD be `204 No Content`. Clients SHOULD send this when aborting ([end-14](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Not covered by upstream OCI conformance suite. | | 33 | `<name>` path validation | HTTP | medium | not started | `<name>` MUST match the regular expression defined in [Pulling Manifests](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pulling-manifests): lowercase alphanumeric segments separated by `.`, `_`, `__`, or `-`, with `/` for nested paths. Not covered by upstream OCI conformance suite. | | 34 | `<reference>` tag validation | HTTP | medium | not started | Tag MUST be at most 128 characters and MUST match `[a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}` ([Pulling Manifests](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pulling-manifests)). Not covered by upstream OCI conformance suite. | | 35 | Manifest stored in exact byte representation | HTTP | medium | not started | Registry MUST store the manifest in the exact byte representation provided by the client ([Pushing Manifests](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pushing-manifests)). Verify by pulling and comparing bytes. Not covered by upstream OCI conformance suite. | | 36 | Empty layer list push | HTTP | medium | not started | The list of blobs MAY be empty ([Pushing Manifests](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pushing-manifests)). Push manifest with no layers; test documents registry behavior. Covered by upstream OCI conformance suite (v1.0.1). | | 37 | Concurrent blob uploads don't corrupt data | HTTP+parallel | high | not started | Two parallel uploads of the same blob must not corrupt stored data. Content-addressable storage means second upload is effectively a no-op. Spec notes: "Even in the case where both uploads are accepted, the registry may securely only store one copy" ([Layer Upload De-duplication](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#layer-upload-de-duplication)). Not covered by upstream OCI conformance suite. | | 38 | Concurrent manifest pushes to same tag | HTTP+parallel | high | not started | Two parallel `PUT` to same tag with different manifests; no corruption or 500 errors. Not covered by upstream OCI conformance suite. | | 39 | Concurrent blob downloads during upload | HTTP+parallel | high | not started | Blob pull while another client is uploading the same blob must not return partial or corrupted data. Either returns the completed blob or `404`. Not covered by upstream OCI conformance suite. | </details> #### Errors <details> <summary>Click to expand</summary> | \# | Test case | Driver | Priority | Status | Notes | |----|-----------|--------|----------|--------|-------| | 1 | 404 for non-existent manifest | HTTP | high | not started | `GET /v2/<name>/manifests/<reference>` for non-existent manifest MUST return `404 Not Found` ([end-3](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 2 | 404 for non-existent blob | HTTP | high | not started | `GET /v2/<name>/blobs/<digest>` for non-existent blob MUST return `404 Not Found` ([end-2](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 3 | 404 for non-existent repository | HTTP | high | not started | `DELETE /v2/<name>/manifests/<digest>` on non-existent repo MUST return `404 Not Found` ([end-9](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Not covered by upstream OCI conformance suite. | | 4 | 401/403 on unauthorized access | HTTP | high | not started | Requests without valid credentials return `401 Unauthorized`. Insufficient permissions return `403` with `DENIED` error code ([code-11](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#error-codes), [code-12](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#error-codes)). Not covered by upstream OCI conformance suite (suite runs without auth). | | 5 | OCI error response format | HTTP | high | not started | Error responses in JSON format MUST have `errors` array with objects containing `code` (uppercase+underscores), optional `message`, optional `detail`. `code` MUST be one of the 14 defined codes ([Error Codes](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#error-codes)). Covered by upstream OCI conformance suite (v1.0.1). | | 6 | `MANIFEST_BLOB_UNKNOWN` on missing references | HTTP | medium | not started | Push manifest referencing non-existent blob(s). When rejected, MUST result in one or more `MANIFEST_BLOB_UNKNOWN` errors ([code-5](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#error-codes)). Not covered by upstream OCI conformance suite. | | 7 | `DIGEST_INVALID` on digest mismatch | HTTP | medium | not started | Upload blob with mismatched digest MUST return `DIGEST_INVALID` ([code-4](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#error-codes)). Not covered by upstream OCI conformance suite. | | 8 | `416 Requested Range Not Satisfiable` for bad chunk range | HTTP | medium | not started | Chunked upload with out-of-order `Content-Range` MUST return `416` ([end-5](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints), [end-6](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints)). Covered by upstream OCI conformance suite (v1.0.1). | | 9 | `UNSUPPORTED` for disabled operations | HTTP | medium | not started | When deletion is disabled, registry MUST respond with `400` or `405`. Error code `UNSUPPORTED` indicates the operation is unsupported ([code-13](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#error-codes)). Not covered by upstream OCI conformance suite. | </details> #### Virtual <details> <summary>Click to expand</summary> | \# | Test case | Driver | Priority | Status | Notes | |----|-----------|--------|----------|--------|-------| | 1 | Upstream resolution order | `crane` | critical | not started | First upstream with the image wins. Not covered by upstream OCI conformance suite. | | 2 | Manifest pull from upstream (pull-through cache) | `crane` | critical | not started | Pull image not in local registry; virtual registry fetches from upstream, caches, and returns it. Not covered by upstream OCI conformance suite. | | 3 | Blob pull from upstream | HTTP | critical | not started | Blob requested from virtual registry is transparently fetched from upstream. Not covered by upstream OCI conformance suite. | | 4 | Tag listing across upstreams | HTTP | high | not started | `GET /v2/<name>/tags/list` merges tags from multiple upstreams. Not covered by upstream OCI conformance suite. | | 5 | Cache behavior and TTL | HTTP | high | not started | Cached manifest served until TTL expires; subsequent pull after TTL triggers upstream re-fetch. Not covered by upstream OCI conformance suite. | | 6 | Cache invalidation on upstream update | HTTP | high | not started | After upstream pushes new tag, virtual registry eventually reflects the update. Not covered by upstream OCI conformance suite. | | 7 | Upstream failure handling | HTTP | high | not started | Healthy upstreams still return results when one upstream is down. Not covered by upstream OCI conformance suite. | | 8 | `ns` query parameter for proxy routing | HTTP | medium | not started | `GET /v2/<name>/manifests/<reference>?ns=<source_host>` routes to correct upstream. Registry that uses `ns` SHOULD return `OCI-Namespace` header in response ([Registry Proxying](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#registry-proxying)). Not covered by upstream OCI conformance suite. | </details> ### Test setup and isolation The tool sets up and tears down its own test environment through the [management API](https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/18458) (`/v1/...`): creating local and virtual repositories, configuring remote upstreams, and setting visibility. The management API serves as test infrastructure, not as a test target. The conformance suite validates client protocols; the management API enables that without external setup. Each test run uses a unique run ID (timestamp or CI job ID) to namespace all created resources (repositories, package versions, image tags), avoiding collisions between parallel runs. Resources are cleaned up at the end of the run via the management API. ### Shared test areas - **Concurrency**: Parallel uploads/downloads, no corruption or lost writes - **Error contracts**: Consistent error response format across formats ### Report format The tool produces a JUnit XML report for CI integration and a human-readable summary to stdout. Each test case is tagged with its format (maven/npm/container) and mode (protocol/auth) so results can be filtered. ## High-Level Implementation Plan <details> <summary>Expand</summary> ### Public Go module ``` gitlab.com/gitlab-org/ops/registry-conformance/ pkg/ client/ maven/ # Maven HTTP protocol client npm/ # npm HTTP protocol client oci/ # OCI distribution client (native, upstream suite as reference) conformance/ # Run(Config) (*Report, error) entry point internal/ cli/ # urfave/cli v3 wiring report/ # JUnit XML + stdout formatting cmd/conformance/main.go docs/ README.md # Index: links to everything below roadmap.md # Test cases, priorities, status (single source of truth) specs/ maven-protocol.md npm-protocol.md oci-protocol.md plans/ # Point-in-time implementation snapshots per MR dev/ getting-started.md ``` `pkg/client/*` and `pkg/conformance` are importable by external consumers. `internal/` stays private. ### Decisions | Decision | Choice | |----------|--------| | Project name | `registry-conformance` | | Public `pkg/` | Yes, importable client library. May be used by AR programmatically as client for internal integration tests. | | OCI strategy | Native implementation, upstream suite as reference only | | Language | Go | | CLI framework | `urfave/cli/v3` | | Build system | Make | | Test framework | `testing` + `testify/require` | | Logging | `slog` | | HTTP | Standard library client | | Report format | JUnit XML + human-readable stdout | | Test isolation | Run-ID namespacing | | Scaffolding | Copier template ([process](https://gitlab.com/gitlab-org/gitlab/-/work_items/591832)) | | Auth (initial) | Valid credentials only, provided by user | | Dev approach | Follows [agentic dev plan](https://docs.google.com/document/d/1MsBmaOlBNzC8uSQcpEnH7b_mkw3zQdnm_slBLR8O0Bk/edit) | ### Phase 0: Test cases, contracts, specs, scaffold (both engineers) #### Step 1: Finalize test cases (parallel) Each engineer picks one format and reviews the test cases from the [issue description](https://gitlab.com/gitlab-org/gitlab/-/work_items/591953) against protocol references. Add missing cases, remove invalid ones, set priorities. Output: `docs/roadmap.md`: ```markdown # Roadmap Status: `not started` | `in progress` | `done` | `blocked` | `skipped` ## Maven | # | Test case | Priority | Status | Notes | |---|-----------|----------|--------|-------| | 1 | Release upload (JAR, POM, checksums, metadata) | critical | not started | | | 2 | Release download (dependency:resolve) | critical | not started | | | 3 | SNAPSHOT upload with timestamped filenames | critical | not started | Build number must increment atomically | | ... | | | | | ## npm | # | Test case | Priority | Status | Notes | |---|-----------|----------|--------|-------| | 1 | Publish unscoped package | critical | not started | PUT /{name} with _attachments | | 2 | Publish scoped package | critical | not started | PUT /@scope%2Fname | | 3 | Install by exact version | critical | not started | | | ... | | | | | ## OCI | # | Test case | Priority | Status | Notes | |---|-----------|----------|--------|-------| | 1 | Blob upload (monolithic) | critical | not started | | | 2 | Manifest PUT | critical | not started | | | 3 | Blob GET | critical | not started | | | ... | | | | | ``` Rows ordered by implementation priority (push before pull, critical before high). Agents update status as they work and read this to determine what to build next. #### Step 2: Align on contracts (together, informed by Step 1) The test cases tell them what methods clients need, what inputs tests require, and what outputs make sense. Define: 1. **Go module API** (`pkg/`): client interfaces per format, `Config` struct, `Report`/`TestCase` types, error types, context propagation, HTTP client injection, run-ID namespacing convention 2. **CLI inputs**: flag names, env var names, config file schema (if any) 3. **CLI outputs**: stdout text summary format, JUnit XML structure, exit codes (0 = pass, 1 = fail, 2 = tool error) Focus on consistent cross-format DX. #### Step 3: Write protocol specs (parallel, grounded in test cases + contracts) Each engineer authors `docs/specs/{format}-protocol.md` for their format: the HTTP contracts for every operation their test cases need (endpoints, methods, headers, request/response bodies, status codes, error behavior), consistent with the agreed API contracts. Sources: official protocol specs (if any), package and container registry implementations, observed client behavior. Specs must be reviewed and approved before implementation begins. They are what agents read when implementing `pkg/client/*`. #### Step 4: Scaffold (together) - Create project via copier template - Commit specs, roadmap, - Set up `CLAUDE.md` (lean index pointing to `docs/`) - CI pipeline: build, lint, test, coverage (should be provided by copier) ### Phase 1: Protocol tests (parallel, both engineers) All tests use valid credentials (auth happy path only). Each engineer takes the format they picked in Step 1. No shared state between formats. One MR per test case. `pkg/client/maven/` replicates Maven client HTTP behavior. Validate against GitLab Package Registry. `pkg/client/npm/` replicates npm client HTTP behavior. Validate against GitLab Package Registry. `pkg/client/oci/` built from scratch using the [OCI conformance suite](https://github.com/opencontainers/distribution-spec/tree/main/conformance) as reference. Validate against Container Registry. Picked up by whoever finishes their format first, or by a third contributor. ### Phase 2: Edge cases and error path (later) - No creds: operations rejected (401) - Invalid creds: operations rejected (401) - ... ### AI-assisted workflow Each phase is designed for burst-style AI-assisted development: 1. Pick the next `not started` test case from `docs/roadmap.md` 2. Agent reads the relevant protocol spec in `docs/specs/` 3. Agent implements (`pkg/client` + conformance tests + CLI wiring) 4. Validate against a live registry 5. Update `docs/roadmap.md` status For MRs that introduce or change test cases, agents write a plan file in `docs/plans/` and the operator approves before implementation. </details> ## Validation Before running against the Artifact Registry, validate the conformance tool against existing implementations to confirm the tests themselves are correct: | Format | Target | Notes | |--------|--------|-------| | Maven | GitLab Package Registry | Rails-based Maven repository in the monolith; validates upload, download, metadata, and checksum tests | | npm | GitLab Package Registry | Rails-based npm registry; validates publish, install, packument, and dist-tag tests | | Container | GitLab Container Registry | Standalone Go service; validates OCI protocol tests alongside the upstream OCI conformance suite | | Virtual (Maven, npm) | GitLab Virtual Registry | Rails-based virtual repositories; validates upstream resolution, caching, and resolution order tests | Since the Artifact Registry does not exist yet, these existing implementations are the only way to validate the test suite. A test that fails against a known-good implementation is a bug in the test, not in the target. Once the suite runs green against these implementations, it becomes the acceptance gate for the Artifact Registry. ## Prior art | Tool | Format | Status | Notes | |------|--------|--------|-------| | [OCI conformance](https://github.com/opencontainers/distribution-spec/tree/main/conformance) | Container | Active, official | Gold standard; standalone binary, structured report | | [`abstract-npm-registry`](https://github.com/godaddy/abstract-npm-registry) | npm | Unmaintained (\~2014) | Mocha-based, covers historical npm API | | [`registry-validator`](https://github.com/strongloop-community/registry-validator) | npm | Unmaintained (\~2014) | CLI tool, publish/fetch validation | | [`repository-validator`](https://github.com/jdcasey/repository-validator) | Maven | Unmaintained | Static consistency, not protocol testing | | [`mh_checkrepo`](https://manpages.debian.org/testing/maven-repo-helper/mh_checkrepo.1.en.html) | Maven | Active (Debian) | Debian repo layout validation only |