test(e2e): the end-to-end program in four profiles, with the scenario catalogs, limitations index and CI arms

The end-to-end program

This branch is the whole e2e effort for Artifact Registry: a suite that drives a deployed registry on a Caproni rig the way a client does — through an ingress, against real PostgreSQL, Valkey and object storage — plus the scenario catalogs it answers to, the tooling that ties the two together, and the CI arms that run it.

193 commits, 339 files, ~63k insertions. It is far too large to review commit by commit, so this description is the shape a reviewer needs in order to judge it.

Four profiles, and what each can prove

A profile is an environment plus what it can prove. Each is a package, each holds only the harness and suite plumbing, and the tests live in surfaces beneath it — one directory, one test binary.

Profile	Environment	Proves	Surfaces
e2e/bootstrap	5 deployers, bootstrap token	the GitLab and management API contracts, and the format protocols	9
e2e/authz	+ monolith, IAM, GLAZ; 11 deployers, 3 Postgres clusters	that a role grant changes an answer	8
e2e/instance	bootstrap rig, Helm release re-applied per surface	that a setting of the running instance changes an answer	5
e2e/benchmark	bootstrap rig	nothing — it measures the read path under go test -bench	1

The split is forced, not stylistic:

• bootstrap cannot assert authorization at all. With no iam block the composition root leaves the container authorization middleware uninstalled, so every authenticated request is allowed everywhere. Any row about a role has to live in authz.
• instance cannot run in parallel. A surface's configuration is carried by the whole deployment, so two overlapping surfaces would each answer under the other's. It runs -p 1 -parallel 1, and its worst outcome is a failed restore: the next surface then asserts against this one's configuration and passes.
• benchmark asserts nothing on purpose. Its numbers are comparable to another run on the same rig shape and to nothing else, so it reports and never gates.

Everything shared sits under e2e/internal/ and is reachable from nowhere else: the kubectl seam, the registry's own HTTP surfaces, derived names, the format clients (grouped by credential-isolation family, not by binary), and the collector, log and metric readers.

One row, one test, one line of a report

The chain has no side table anyone maintains:

docs/testing/e2e/*.md        a row declares the scenario and its id
  └─ e2e-catalog generate →  one Go constant per row (a constant cannot name a row that does not exist)
       └─ a test runs its body as t.Run(scenarioid.X, …)  → the id is the subtest name
            └─ gotestsum --junitfile → e2e-reports/junit-<profile>.xml
                 └─ e2e-catalog report → catalog-report.md, joined back to the catalogs

e2e-catalog check walks it in both directions and fails on either kind of lie: a Status claiming implemented with no test, and a test declaring a row the catalog calls not started.

What a green suite does not attest

This is the part worth a reviewer's attention, because it is where a test suite usually overclaims.

docs/testing/e2e/limitations.md is a single index of every limitation the suite runs under, filed under the change that would lift it — 1. Harness and CI (1 entry), 2. Caproni rig (7), 3. Third-party features (6), 4. Artifact Registry backend (12). Two of its tables are machine-checked:

• Rows no test can assert — a row out of reach whatever the effort. check reports it if a test later declares it, which is how a lifted limitation surfaces.
• Green does not attest — a row that is covered, with the clause its pass does not prove. This exists so a row is written and declared rather than withheld: a withheld row is neither covered nor registered, and reports as a disagreement forever.

The rig differs from production in ways the index records rather than hides: it serves plain HTTP, the docker rows address a port-forward because no Traefik entrypoint can serve them, iam.cache_ttl is 0s where production caches, and one organization exists where the model allows many.

Current state

Measured at 10d34ed22, which is what these figures are true at: go run ./e2e/cmd/e2e-catalog check. They drift from the next commit on, so a reader who re-runs it and gets a different number is looking at a later tree rather than a wrong command.

Scenarios	231 (docker 71, oci 70, maven 72, npm 126, management 13 — counting both tables per file)
Covered	181
Blocked / manual	30 / 1
Registered unassertable	14
Covered with a recorded caveat	4
Backlog	5

All 5 backlog rows arrived with main in the merge below, checked one by one rather than inferred: e2e.docker.discover.manifest-detail-via-api, e2e.oci.discover.manifest-detail-via-api, e2e.npm.publish.stalled-publish-cut, e2e.npm.publish.provenance-absent-without-flag and e2e.npm.publish.reject-wrong-second-attachment. Each is declared in main and absent from this branch's tip before the merge, and none is covered here. lint:e2e-catalog-sync is therefore red and allow_failure: true — it reports the backlog rather than blocking on it. That red is the honest state and is not suppressed.

CI

Five jobs, all extending one template so the runner, the two docker:dind services, the mise bootstrap, the diagnostics and the artifacts cannot drift:

Job	Runs
test:e2e:caproni:bootstrap	45m
test:e2e:caproni:authz	90m
test:e2e:caproni:instance	45m, serial
test:e2e:caproni:benchmark	45m, reports rather than gates
test:e2e:catalog-report	joins the JUnit to the catalogs

Each is gated three ways: automatic on its own RUN_E2E_* schedule variable; automatic and blocking on this branch, which is what makes this merge request's own pipeline the proof that the suite is green; and manual, non-blocking everywhere else, so a change to the harness can be exercised without holding anyone's pipeline.

Automatic and blocking together is deliberate. allow_failure is what makes a job non-blocking, so an automatic job that kept it would run the whole rig and leave a red result under a green pipeline.

The branch has an open merge request — this one — and workflow:rules refuses a branch pipeline while one exists, so every pipeline this branch has ever had is this merge request's. The rules key on the merge request's source branch for that reason, with a CI_COMMIT_BRANCH arm behind it for after this lands.

Reviewing it

Read in this order. Each answers the question the next one assumes.

1. e2e/README.md — the package-level document: the layout, the dependency direction, why names are derived rather than random, how a grant reaches IAM through Rails, and the traps each seam encodes.
2. docs/testing/e2e/README.md — the catalog conventions, and how to read a red lint:e2e-catalog-sync.
3. docs/testing/e2e/limitations.md — what the suite cannot do, and what a green result therefore does not mean.
4. docs/dev/caproni-e2e.md — the operator's view: what each environment deploys, what it omits, how to run it, and how CI runs it.

Then a single surface end to end — e2e/bootstrap/oras/consume_test.go is the smallest complete example — and the profile plumbing it rests on.

main is merged in as of this branch's tip rather than rebased: two merge requests were targeting this branch while the work was in flight, and a rebase would have orphaned them. Two files conflicted and neither took a side; the merge commit records the resolution and the row-set diff against both parents that proves nothing was dropped.

Related to #949