feat(maven): batched hosted lookups and their adapter (S30 plan: 5/25)

Moaz Khalifarequested to merge
mkhalifa3/s30-maven-virtual-step-5 into main

Summary

Implements Step 5 of the S30 plan: the batched authoritative read over the hosted Maven tables, and the Phase 1 remote.Upstream adapter that serves a virtual position's verdict from it. Depends on: — per the plan, so it targets main directly.

MavenVirtualHostedLookupStore runs one statement per document class — the package-level stored maven-metadata.xml and the version-level file — over maven_files joined to its maven_packages and maven_versions parents, with namespace_id equality on every joined table, soft_deleted_at IS NULL on all three, and the positions as one IN list. The package-level statement also reports, in that same statement, whether the package holds any version-level files: the source of the merge path's no_stored_document outcome. VirtualHostedUpstreams hands each position its prefetched verdict — a Miss is an absence the hosted upstream established, a Hit is always fresh — and errors on every fault no signature rules out: an undefined target layout, a hit outside the requested batch, a short blob_sha256, a nil store.

Reviewable LOC: 2,732 (2,728 insertions, 4 deletions) — past the 500 ceiling. By file group: the store and its query names 419, the adapter 255, tests 2,054. A split would not help: the two source files are one seam's two halves — the adapter's every arm consumes the store's two reads, so its MR would be a stacked dependency on a 419-line first part with no consumer. The nearest precedent split smaller: npm's own split of this step produced parts of 1,170 and 1,234 whole-diff LOC (!1974 (merged)), so the size rule alone would argue for splitting here too; the seam argument, not the size, is why this step ships whole.

References: S30 spec · S30 plan (see "Step 5: hosted upstream lookup")

Governing ADRs

  • ADR-001 (organizations as anchor) — namespace_id equality leads on every joined table in both reads and in the presence-check EXISTS; the integration suite asserts cross-namespace positions are unreachable.
  • ADR-004 (data and application limits) — the reads enforce no limit and bypass none; the 20-upstream cap is cited as the batch's bound.
  • ADR-005 (artifact delivery mode) — the adapter answers verdicts only; the redirect-or-proxy decision belongs to the serve path of later steps.
  • ADR-007 (database schema) — namespace-leading predicates and soft_deleted_at IS NULL on all three tables match the ADR's Maven query examples; the maven_version_id IS NULL / IS NOT NULL split matches the two partial unique indexes, and the EXPLAIN guard pins single-partition pruning per table.
  • ADR-008 (content-addressable storage) — format checksums stay on maven_files; the blob ref validates as a 32-byte SHA-256 before it becomes a digest. No storage-layer decision is made here.
  • ADR-009 (API design) — no route, endpoint, or URL shape added or altered.
  • ADR-012 (usage data) — no usage events emitted; the virtual pull stream is Step 17's.
  • Mirror freshness: scripts/adr-freshness.sh reports upstream amendments to ADR-007 and ADR-022 not yet synced; the upstream ADR-007 delta touches container index listings only, nothing this step relies on.

Testing

Mirrors npm's step-5 suites per the mirroring rule, with no dropped subtest that has a Maven analog:

  • Statement-rendering unit tests assert the authoritative-row projection (no maven_remote_ table in either statement), the namespace-literal predicates, and the soft-delete gates.
  • The guards suite rejects every bind that would silently read as absence — zero UUIDs, empty coordinates, nil context, nil client — with its own sentinel.
  • The integration suite covers both document classes: hits, established misses, soft-delete negatives per document class at every level of the chain, cross-namespace isolation, one statement per class at 2 and at 20 positions, and error-not-Miss when the read fails.
  • The EXPLAIN guard pins single-partition pruning per table, the exact planned-table set, an unseeded namespace, and twenty positions.
  • The adapter suite pins the zero-value Miss, the fresh-Hit contract, the per-document-class one-read property, and store-error propagation.

Integration suite run locally against a CI-tuned PostgreSQL; golangci-lint run with --build-tags=integration and the documented flag set, which CI lint cannot see. After the rebase onto today's main: go test ./internal/datastore/ ./internal/format/maven/ green.

A five-category pre-push coherence check (plan adherence, acceptance criteria, spec adherence, ADR adherence, dev guidelines) ran on this branch: four PASS, one WARN, closed by the two advisory comment fixes that same push landed.

e2e catalog

No scenario added or affected. This step adds no route and no handler: nothing dispatches to these reads until the virtual dispatch arm (Step 9) and the composition root (Step 15) land, and the virtual e2e scenarios are Step 20's deliverable.

Database Review Evidence

Queries

Note

Plans are from EXPLAIN (ANALYZE, BUFFERS) against an ephemeral PostgreSQL 17 container (matching GL_PG_CURR_VERSION from .gitlab-ci-other-versions.yml), with synthesized seed data rolled back per query and the container torn down at the end of the run. Numbers reflect moderate cardinality and do not capture production-scale effects. See Database review evidence for seed sizing, methodology, and the anomalies the skill flags. Expand each row's details for the seed shape, rendered SQL, bound args, and raw plan.

Method Plan node Index Rows (plan / actual) Cost Time Buffers (hit / read) Partitions
datastore.MavenVirtualHostedPackageDocumentLookup Nested Loop Left Join maven_packages_p57_namespace_id_maven_repository_id_group_i_idx, maven_files_p57_namespace_id_maven_package_id_file_name_idx, maven_versions_p57_namespace_id_maven_package_id_id_idx, maven_files_p57_namespace_id_maven_version_id_file_name_idx 4 / 2 97.46 0.075ms 23 / 0 1/64, 1/64, 1/64
datastore.MavenVirtualHostedVersionFileLookup Nested Loop maven_packages_p57_namespace_id_maven_repository_id_group_i_idx, maven_versions_p57_namespace_id_maven_package_id_size_bytes_idx, maven_files_p57_namespace_id_maven_version_id_file_name_idx 1 / 2 47.62 0.037ms 17 / 0 1/64, 1/64, 1/64
datastore.MavenVirtualHostedPackageDocumentLookup

Summary: Plan matches the method's intent: the composite (namespace_id, maven_repository_id, group_id, artifact_id) index seeks the two requested positions' live packages, the LEFT JOIN reads each position's stored package-level document from the version-level-NULL partial index on (namespace_id, maven_package_id, file_name), and the correlated holds_version_files EXISTS rides two index-only scans. Every partitioned table prunes to 1/64 partitions, actual rows (2) equal the two requested positions, and the result set is bounded by the caller-capped position IN-list against unique-column lookups. No anomalies.

Seed shape: namespaces=1, repositories=2, maven_repositories=2, maven_packages=5000, maven_versions=5000, blob_storage_blobs=9886, blob_storage_attachments=9886, maven_files=9886 (5000 documents + 4886 version-level)

Rendered SQL:

SELECT maven_packages.maven_repository_id AS "maven_packages.maven_repository_id",
     maven_files.namespace_id AS "maven_files.namespace_id",
     maven_files.id AS "maven_files.id",
     maven_files.maven_package_id AS "maven_files.maven_package_id",
     maven_files.maven_version_id AS "maven_files.maven_version_id",
     maven_files.blob_storage_attachment_id AS "maven_files.blob_storage_attachment_id",
     maven_files.soft_deleted_at AS "maven_files.soft_deleted_at",
     maven_files.blob_sha256 AS "maven_files.blob_sha256",
     maven_files.sha1 AS "maven_files.sha1",
     maven_files.sha512 AS "maven_files.sha512",
     maven_files.md5 AS "maven_files.md5",
     maven_files.file_name AS "maven_files.file_name",
     maven_files.created_at AS "maven_files.created_at",
     (EXISTS (
          SELECT maven_versions.id AS "maven_versions.id"
          FROM public.maven_versions
               INNER JOIN public.maven_files ON ((maven_files.namespace_id = maven_versions.namespace_id) AND (maven_files.maven_version_id = maven_versions.id))
          WHERE ((((maven_versions.namespace_id = $1::uuid) AND (maven_files.namespace_id = $2::uuid)) AND (maven_versions.maven_package_id = maven_packages.id)) AND (maven_versions.soft_deleted_at IS NULL)) AND (maven_files.soft_deleted_at IS NULL)
     )) AS "presence.holds_version_files"
FROM public.maven_packages
     LEFT JOIN public.maven_files ON (((((maven_files.namespace_id = $3::uuid) AND (maven_files.maven_package_id = maven_packages.id)) AND (maven_files.maven_version_id IS NULL)) AND (maven_files.file_name = $4::text)) AND (maven_files.soft_deleted_at IS NULL))
WHERE ((((maven_packages.namespace_id = $5::uuid) AND (maven_packages.maven_repository_id IN ($6::uuid, $7::uuid))) AND (maven_packages.group_id = $8::text)) AND (maven_packages.artifact_id = $9::text)) AND (maven_packages.soft_deleted_at IS NULL);

Bound args: [namespace 00000000-0000-7000-8000-000000000001 (EXISTS maven_versions), namespace 00000000-0000-7000-8000-000000000001 (EXISTS maven_files), namespace 00000000-0000-7000-8000-000000000001 (LEFT JOIN maven_files), 'maven-metadata.xml', namespace 00000000-0000-7000-8000-000000000001 (maven_packages), position 00000000-0000-7000-8000-0000000000a1, position 00000000-0000-7000-8000-0000000000a2, 'org.apache.maven', 'maven-artifact']

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Nested Loop Left Join  (cost=0.56..97.46 rows=4 width=276) (actual time=0.039..0.049 rows=2 loops=1)
   Buffers: shared hit=23
   ->  Index Scan using maven_packages_p57_namespace_id_maven_repository_id_group_i_idx on maven_packages_p57 maven_packages  (cost=0.28..14.36 rows=2 width=32) (actual time=0.012..0.015 rows=2 loops=1)
         Index Cond: ((namespace_id = '00000000-0000-7000-8000-000000000001'::uuid) AND (maven_repository_id = ANY ('{00000000-0000-7000-8000-0000000000a1,00000000-0000-7000-8000-0000000000a2}'::uuid[])) AND (group_id = 'org.apache.maven'::text) AND (artifact_id = 'maven-artifact'::text))
         Buffers: shared hit=5
   ->  Index Scan using maven_files_p57_namespace_id_maven_package_id_file_name_idx on maven_files_p57 maven_files  (cost=0.28..8.30 rows=1 width=259) (actual time=0.004..0.004 rows=1 loops=2)
         Index Cond: ((namespace_id = '00000000-0000-7000-8000-000000000001'::uuid) AND (maven_package_id = maven_packages.id) AND (file_name = 'maven-metadata.xml'::text))
         Buffers: shared hit=6
   SubPlan 1
     ->  Nested Loop  (cost=0.56..16.62 rows=1 width=0) (actual time=0.012..0.012 rows=1 loops=2)
           Buffers: shared hit=12
           ->  Index Only Scan using maven_versions_p57_namespace_id_maven_package_id_id_idx on maven_versions_p57 maven_versions  (cost=0.28..8.30 rows=1 width=32) (actual time=0.006..0.006 rows=1 loops=2)
                 Index Cond: ((namespace_id = '00000000-0000-7000-8000-000000000001'::uuid) AND (maven_package_id = maven_packages.id))
                 Heap Fetches: 2
                 Buffers: shared hit=6
           ->  Index Only Scan using maven_files_p57_namespace_id_maven_version_id_file_name_idx on maven_files_p57 maven_files_1  (cost=0.28..8.30 rows=1 width=32) (actual time=0.006..0.006 rows=1 loops=2)
                 Index Cond: ((namespace_id = '00000000-0000-7000-8000-000000000001'::uuid) AND (maven_version_id = maven_versions.id))
                 Heap Fetches: 2
                 Buffers: shared hit=6
 Planning:
   Buffers: shared hit=1048 read=1
 Planning Time: 2.098 ms
 Execution Time: 0.075 ms

Timings: planning 2.098ms, execution 0.075ms, total 2.173ms.

datastore.MavenVirtualHostedVersionFileLookup

Summary: Plan matches the method's intent: three nested-loop index scans — packages via the coordinate index, versions via the (namespace_id, maven_package_id, size_bytes) index with the version predicate as a post-scan filter, and files via the version-level partial index carrying file_name as an index condition. All three tables prune to 1/64 partitions, actual rows (2) equal the two requested positions, and the result set is bounded by the caller-capped position IN-list against unique-column lookups. The version filter rides the FK-support index rather than the partial (namespace_id, maven_package_id, version) unique index — one row per package either way, no cost difference at this cardinality. No anomalies.

Seed shape: namespaces=1, repositories=2, maven_repositories=2, maven_packages=5000, maven_versions=5000, blob_storage_blobs=9886, blob_storage_attachments=9886, maven_files=9886 (5000 documents + 4886 version-level)

Rendered SQL:

SELECT maven_packages.maven_repository_id AS "maven_packages.maven_repository_id",
     maven_files.namespace_id AS "maven_files.namespace_id",
     maven_files.id AS "maven_files.id",
     maven_files.maven_package_id AS "maven_files.maven_package_id",
     maven_files.maven_version_id AS "maven_files.maven_version_id",
     maven_files.blob_storage_attachment_id AS "maven_files.blob_storage_attachment_id",
     maven_files.soft_deleted_at AS "maven_files.soft_deleted_at",
     maven_files.blob_sha256 AS "maven_files.blob_sha256",
     maven_files.sha1 AS "maven_files.sha1",
     maven_files.sha512 AS "maven_files.sha512",
     maven_files.md5 AS "maven_files.md5",
     maven_files.file_name AS "maven_files.file_name",
     maven_files.created_at AS "maven_files.created_at"
FROM public.maven_files
     INNER JOIN public.maven_versions ON ((maven_versions.namespace_id = maven_files.namespace_id) AND (maven_versions.id = maven_files.maven_version_id))
     INNER JOIN public.maven_packages ON ((maven_packages.namespace_id = maven_versions.namespace_id) AND (maven_packages.id = maven_versions.maven_package_id))
WHERE (((((((((((maven_packages.namespace_id = $1::uuid) AND (maven_packages.maven_repository_id IN ($2::uuid, $3::uuid))) AND (maven_packages.group_id = $4::text)) AND (maven_packages.artifact_id = $5::text)) AND (maven_packages.soft_deleted_at IS NULL)) AND (maven_versions.namespace_id = $6::uuid)) AND (maven_versions.version = $7::text)) AND (maven_versions.soft_deleted_at IS NULL)) AND (maven_files.namespace_id = $8::uuid)) AND (maven_files.file_name = $9::text)) AND (maven_files.maven_version_id IS NOT NULL)) AND (maven_files.soft_deleted_at IS NULL);

Bound args: [namespace 00000000-0000-7000-8000-000000000001 (maven_packages), position 00000000-0000-7000-8000-0000000000a1, position 00000000-0000-7000-8000-0000000000a2, 'org.apache.maven', 'maven-artifact', namespace 00000000-0000-7000-8000-000000000001 (maven_versions), '1.0.0', namespace 00000000-0000-7000-8000-000000000001 (maven_files), 'maven-artifact-1.0.0.jar']

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Nested Loop  (cost=0.85..47.62 rows=1 width=275) (actual time=0.017..0.023 rows=2 loops=1)
   Buffers: shared hit=17
   ->  Nested Loop  (cost=0.56..30.99 rows=2 width=48) (actual time=0.012..0.017 rows=2 loops=1)
         Buffers: shared hit=11
         ->  Index Scan using maven_packages_p57_namespace_id_maven_repository_id_group_i_idx on maven_packages_p57 maven_packages  (cost=0.28..14.36 rows=2 width=48) (actual time=0.005..0.007 rows=2 loops=1)
               Index Cond: ((namespace_id = '00000000-0000-7000-8000-000000000001'::uuid) AND (maven_repository_id = ANY ('{00000000-0000-7000-8000-0000000000a1,00000000-0000-7000-8000-0000000000a2}'::uuid[])) AND (group_id = 'org.apache.maven'::text) AND (artifact_id = 'maven-artifact'::text))
               Buffers: shared hit=5
         ->  Index Scan using maven_versions_p57_namespace_id_maven_package_id_size_bytes_idx on maven_versions_p57 maven_versions  (cost=0.28..8.30 rows=1 width=48) (actual time=0.004..0.004 rows=1 loops=2)
               Index Cond: ((namespace_id = '00000000-0000-7000-8000-000000000001'::uuid) AND (maven_package_id = maven_packages.id))
               Filter: (version = '1.0.0'::text)
               Buffers: shared hit=6
   ->  Index Scan using maven_files_p57_namespace_id_maven_version_id_file_name_idx on maven_files_p57 maven_files  (cost=0.28..8.30 rows=1 width=259) (actual time=0.003..0.003 rows=1 loops=2)
         Index Cond: ((namespace_id = '00000000-0000-7000-8000-000000000001'::uuid) AND (maven_version_id = maven_versions.id) AND (file_name = 'maven-artifact-1.0.0.jar'::text))
         Buffers: shared hit=6
 Planning:
   Buffers: shared hit=23
 Planning Time: 0.348 ms
 Execution Time: 0.037 ms

Timings: planning 0.348ms, execution 0.037ms, total 0.385ms.

Related to #289

Edited by Moaz Khalifa

Merge request reports