Draft: feat(pypi): implement S34 PyPI hosted format

Suleimi Ahmedrequested to merge
suleimiahmed/pypi into main

What does this MR do and why?

Implements the S34 PyPI hosted format end to end, per the spec and plan: Python developers can twine/poetry/uv publish and pip/poetry/uv install from a hosted PyPI repository, with the Simple Repository API (HTML + JSON), PEP 658 metadata sidecars, and hash-checking installs.

The 22 plan steps land as 22 single, dependency-ordered commits — each one clean, self-contained, and independently reviewable (and bisectable). Review commit-by-commit.

Stacking / merge order: this branch stacks on the spec MR !1007 and the plan MR !1009 (neither merged yet), so the diff against main currently includes the spec and plan files. Merge order: !1007 → !1009 → this MR; the diff reduces to the implementation once the ancestors land.

What's implemented (by commit)

  • Schema (pypi_* tables, partitioned, UUIDv7, composite keys; append-only filename-claims ledger) + format-enum registration.
  • Config PypiConfig (proto + loader + config.example.yaml + configuration-reference).
  • Validation cores: PEP 503 name normalization, PEP 440 version parse/canonicalize, wheel/sdist filename grammars + coherence, stored-metadata allow-list.
  • Extraction: ranged-read core-metadata extraction (wheels + sdists) behind a decompression guard (stored/deflated, 64 MiB, ratio 50).
  • Datastore: read queries, publish-side committer (one transaction, RETURNING-gated counters, race-free claims + one-sdist guards, unwrapped FK sentinels), deletion-semantics store methods.
  • HTTP: transport (S01 envelope, closed code set), resolver/middleware, route skeleton + Basic-auth dispatch, Simple index renderers + endpoints (Accept negotiation, strong ETags, redirects), download + .metadata endpoints, the three-stage publish flow (parse/pre-check → stream/digests/extraction → commit/respond).
  • Observability (in-handler metrics + wide events), real-handler wiring (composition root, RegisterMetrics), and the conformance harness (scripts/conformance/pypi-e2e.sh driving pip/twine/poetry/uv, conformance:pypi:s3-garage hard gate) + the docs/testing/e2e/pypi.md catalog.

Validation

Every step was authored test-first and adversarially validated. All 48 spec acceptance criteria have genuine covering tests. Local CI parity is green: go build, unit + integration tests, golangci-lint (v2.12, 0 issues), squawk, buf lint + generate-check, jet generate-check, markdownlint + vale (changed docs), integration-test-wiring, depguard negative-control, and -race.

Known items / follow-ups

  • Spec/plan wording touch-ups belong on !1007/!1009 (ratio-floor clause, blob-FK wording, versions_count-vs-deletion note, fuzz-file note).
  • External: the format::pypi scoped label does not exist yet; the Snowplow artifact_registry_context iglu schema must accept pypi; ADR 004/007/009 amendments (handbook).
  • PEP 740 attestations deferred (#363).

Related to #365

Merge request reports