feat(datastore): maven remote cache backing store (S14 Step 8, part 1/2)

Moaz Khalifarequested to merge
mkhalifa/maven-remote-step-8-part1of2 into main

What

Adds the coordinate-based backing store for the Maven remote cache — Step 8, part 1/2 of the S14 Maven remote vertical slice (the split of feat(maven): remote cache store Lookup + freshness per the 500-LOC MR guideline; part 1/2 is the datastore half, part 2/2 the format wrapper stacked on this branch).

internal/datastore/maven_remote_cache.go — MavenRemoteCacheStore: resolves maven_remote_files rows through the package and version joins (version-level by (maven_remote_version_id, file_name), package-level by (maven_remote_package_id, file_name) with a NULL version), with the soft-delete gate applied at every level. Returns all four digest columns (blob_sha256, sha1, md5, sha512) plus the freshness inputs, so the serve and sidecar paths need no second query. The backing store is persistence-only: it computes no freshness verdict — the wrapper owns it through remote.IsFresh (internal/remote's seam contract), so the stored row and the verdict cannot come from different clocks. Rejects nil context, zero namespace, zero repository, and empty coordinates with the maven-remote sentinels before any query runs.

Part 2/2 (internal/format/maven/remote_store.go, the path-parsing remote.CacheStore wrapper) stacks on this branch: the datastore package cannot import internal/format/ (ADR-023), so the wrapper owns parsing and freshness selection and delegates row resolution here.

Coverage

  • internal/datastore/maven_remote_cache_integration_test.go (integration) — coordinate resolution for version- and package-level rows, every projected column asserted with non-zero fixtures, soft-delete misses at all four levels, cross-repository isolation, and the argument guards.
  • internal/datastore/maven_seed_integration_test.go — the randMD5 fixture helper the projection assertions share.
  • Freshness-verdict coverage lives on the wrapper's side (part 2/2, internal/format/maven/remote_store_integration_test.go): Lookup's remote.IsFresh verdicts — fresh/stale windows, the 0 pin, and the snapshot-metadata force-stale override — with the datastore row deliberately verdict-free.

Plan amendment folded in

The plan-file edits (the Research Findings correction recording the wrapper/backing split, the Step 8/9 Files and Tests updates, the Step 13 per-request wiring note, and the Step 8 size record) ride with this MR by operator decision, deviating from the separate plan-amendment MR rule in docs/dev/agentic-development.md. The plan amendment describes the full two-tier split; part 2/2 completes it.

Size note

Part 1/2 is ~840 Go LOC. The overage against the plan's ~400 estimate is, per the plan's Step 8 size row: the integration suite's seed/assert boilerplate and the argument-guard and non-zero-projection coverage the review guidelines demand. Part-1 review removed the backing store's SQL-computed freshness verdict (the wrapper owns freshness through remote.IsFresh), which dropped the SQL-rendering unit suite and the backing store's validity parameters along with it.

Validation

Re-ran the step-8 validation after addressing every finding of the first pass; all five categories passed (plan adherence, acceptance criteria, spec adherence, ADR adherence, dev-guideline adherence). This MR carries the datastore half of those fixes; the verdicts for this half: spec PASS, acceptance PASS, ADR PASS, dev guidelines PASS (constructor panics name the constructor, every projected column seeded non-zero and asserted, subtests use tt).

Database Review Evidence

Queries

Note

Plans are from EXPLAIN (ANALYZE, BUFFERS) against an ephemeral PostgreSQL 17 container (matching GL_PG_CURR_VERSION from .gitlab-ci-other-versions.yml), with synthesized seed data rolled back per query and the container torn down at the end of the run. The dense profile is reported — the read target at ~6,000 rows and the joined parents (maven_remote_packages, maven_remote_versions) at ~5,100 rows in the same partition — which is what the plan looks like at production cardinality. The seed uses many versions per package (100 under the target package) and many files per version (10 each) so the planner's index choice between the partial uniques and the non-partial fallback is exercised, not masked by 1:1:1 fan-out. See Database review evidence for seed sizing, methodology, and the anomalies the skill flags. Expand each row's details for the seed shape, rendered SQL, bound args, and raw plan.

Method Plan node Index Rows (plan / actual) Cost Time Buffers (hit / read) Partitions
datastore.MavenRemoteCacheStore.LookupRow.VersionLevel Limit maven_remote_packages_p35_…_reposit_idx, maven_remote_versions_p35_…_packag_idx3, maven_remote_files_p35_…_version_i_idx1 1 / 1 17.12 0.070ms 10 / 0 1
datastore.MavenRemoteCacheStore.LookupRow.PackageLevel Limit maven_remote_files_p35_…_package_id_idx (partial unique), maven_remote_packages_p35_…_reposit_idx 1 / 1 16.46 0.019ms 5 / 0 1
datastore.MavenRemoteCacheStore.LookupRow.VersionLevel

Summary: Plan matches the method's intent and is index-driven end to end at production cardinality: the package is resolved through the unique full-key index (namespace_id, maven_remote_repository_id, group_id, artifact_id) (all four keys in the Index Cond), the version through the (namespace_id, maven_remote_package_id) index with version as a residual filter that removes 99 of 100 seeded versions (the fan-out the dense seed exposes), and the file through the (namespace_id, maven_remote_version_id) index with maven_remote_version_id IS NOT NULL in the Index Cond (the guard this MR added) and file_name + soft_deleted_at IS NULL as residual filters. The planner chose the non-partial index_maven_remote_files_on_ns_id_ver_id over the partial unique unique_maven_remote_files_ns_id_ver_id_file_name — both prune to the same 10 files for the matched version, and the narrower non-partial index wins on cost. The IS NOT NULL guard is visible in the Index Cond, so a predicate change that dropped it would change the plan shape. With 5,001 packages, 5,100 versions, and 6,002 files in the same partition, the planner prefers all three indexes over Seq Scan; namespace_id prunes to one of 64 partitions, plan rows match actual (1 / 1), and execution stays under 100µs. No anomalies.

Seed shape: namespaces=1, repositories=1, maven_remote_repositories=1, blob_storage_blobs=1, blob_storage_attachments=1, maven_remote_packages=5001, maven_remote_versions=5100, maven_remote_files=6002

The target package (com.example:myapp) has 100 versions (99 decoys + 1.0), 10 files per version (1,000 version-level files), 1 version-level maven-metadata.xml under 1.0, and 1 package-level maven-metadata.xml (NULL version). 5,000 decoy packages each have 1 version and 1 file.

Rendered SQL:

SELECT maven_remote_files.id AS "maven_remote_files.id",
     maven_remote_files.blob_storage_attachment_id AS "maven_remote_files.blob_storage_attachment_id",
     maven_remote_files.blob_sha256 AS "maven_remote_files.blob_sha256",
     maven_remote_files.sha1 AS "maven_remote_files.sha1",
     maven_remote_files.md5 AS "maven_remote_files.md5",
     maven_remote_files.sha512 AS "maven_remote_files.sha512",
     maven_remote_files.upstream_checked_at AS "maven_remote_files.upstream_checked_at",
     maven_remote_files.upstream_etag AS "maven_remote_files.upstream_etag"
FROM public.maven_remote_files
     INNER JOIN public.maven_remote_versions ON ((maven_remote_versions.id = maven_remote_files.maven_remote_version_id) AND (maven_remote_versions.namespace_id = maven_remote_files.namespace_id))
     INNER JOIN public.maven_remote_packages ON ((maven_remote_packages.id = maven_remote_versions.maven_remote_package_id) AND (maven_remote_packages.namespace_id = maven_remote_versions.namespace_id))
WHERE (((((((((((maven_remote_files.namespace_id = $1::uuid) AND (maven_remote_versions.namespace_id = $2::uuid)) AND (maven_remote_packages.namespace_id = $3::uuid)) AND (maven_remote_files.maven_remote_version_id IS NOT NULL)) AND (maven_remote_packages.maven_remote_repository_id = $4::uuid)) AND (maven_remote_packages.group_id = $5::text)) AND (maven_remote_packages.artifact_id = $6::text)) AND (maven_remote_versions.version = $7::text)) AND (maven_remote_files.file_name = $8::text)) AND (maven_remote_files.soft_deleted_at IS NULL)) AND (maven_remote_versions.soft_deleted_at IS NULL)) AND (maven_remote_packages.soft_deleted_at IS NULL))
LIMIT $9;

Bound args: [<namespace-uuid>, <namespace-uuid>, <namespace-uuid>, <maven-remote-repository-uuid>, com.example, myapp, 1.0, file-01.jar, 1]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Limit  (cost=0.85..17.12 rows=1 width=215) (actual time=0.039..0.039 rows=1 loops=1)
   Buffers: shared hit=10
   ->  Nested Loop  (cost=0.85..17.12 rows=1 width=215) (actual time=0.038..0.039 rows=1 loops=1)
         Buffers: shared hit=10
         ->  Nested Loop  (cost=0.56..16.62 rows=1 width=32) (actual time=0.026..0.026 rows=1 loops=1)
               Buffers: shared hit=7
               ->  Index Scan using maven_remote_packages_p35_namespace_id_maven_remote_reposit_idx on maven_remote_packages_p35 maven_remote_packages  (cost=0.28..8.31 rows=1 width=32) (actual time=0.009..0.009 rows=1 loops=1)
                     Index Cond: ((namespace_id = '<ns>'::uuid) AND (maven_remote_repository_id = '<mrr>'::uuid) AND (group_id = 'com.example'::text) AND (artifact_id = 'myapp'::text))
                     Buffers: shared hit=3
               ->  Index Scan using maven_remote_versions_p35_namespace_id_maven_remote_packag_idx3 on maven_remote_versions_p35 maven_remote_versions  (cost=0.28..8.30 rows=1 width=48) (actual time=0.016..0.016 rows=1 loops=1)
                     Index Cond: ((namespace_id = '<ns>'::uuid) AND (maven_remote_package_id = maven_remote_packages.id))
                     Filter: ((soft_deleted_at IS NULL) AND (version = '1.0'::text))
                     Rows Removed by Filter: 99
                     Buffers: shared hit=4
         ->  Index Scan using maven_remote_files_p35_namespace_id_maven_remote_version_i_idx1 on maven_remote_files_p35 maven_remote_files  (cost=0.28..0.49 rows=1 width=247) (actual time=0.011..0.011 rows=1 loops=1)
               Index Cond: ((namespace_id = '<ns>'::uuid) AND (maven_remote_version_id = maven_remote_versions.id) AND (maven_remote_version_id IS NOT NULL))
               Filter: ((soft_deleted_at IS NULL) AND (file_name = 'file-01.jar'::text))
               Buffers: shared hit=3
 Planning:
   Buffers: shared hit=456 read=1
 Planning Time: 1.358 ms
 Execution Time: 0.070 ms

Timings: planning 1.358ms, execution 0.070ms, total 1.428ms.

datastore.MavenRemoteCacheStore.LookupRow.PackageLevel

Summary: Plan matches the method's intent and is index-driven at production cardinality: the file is resolved through the partial unique index unique_maven_remote_files_ns_id_pkg_id_file_name (namespace_id, maven_remote_package_id, file_name) WHERE soft_deleted_at IS NULL AND maven_remote_version_id IS NULL — the maven_remote_version_id IS NULL guard and soft_deleted_at IS NULL gate are both covered by the partial predicate, so neither appears as a Filter. The namespace_id and file_name constants are in the Index Cond; maven_remote_package_id is resolved through the join. The package is then resolved through the unique full-key index (namespace_id, maven_remote_repository_id, group_id, artifact_id). Both namespace_id literals prune to one of 64 partitions. With 5,001 packages and 6,002 files in the same partition (1 package-level maven-metadata.xml and 1,001 version-level files under the target package), the partial unique scans only the 1 package-level row — a non-partial index would have scanned all 1,002 files under the package. Plan rows match actual (1 / 1) and execution stays under 20µs. No anomalies.

Seed shape: namespaces=1, repositories=1, maven_remote_repositories=1, blob_storage_blobs=1, blob_storage_attachments=1, maven_remote_packages=5001, maven_remote_files=6002

Same seed as the version-level query. The target package (com.example:myapp) has 1 package-level maven-metadata.xml (NULL version_id) and 1,001 version-level files (1,000 + 1 version-level maven-metadata.xml). The query asks for the package-level file, so the IS NULL guard directs the planner to the partial unique that contains only the 1 package-level row.

Rendered SQL:

SELECT maven_remote_files.id AS "maven_remote_files.id",
     maven_remote_files.blob_storage_attachment_id AS "maven_remote_files.blob_storage_attachment_id",
     maven_remote_files.blob_sha256 AS "maven_remote_files.blob_sha256",
     maven_remote_files.sha1 AS "maven_remote_files.sha1",
     maven_remote_files.md5 AS "maven_remote_files.md5",
     maven_remote_files.sha512 AS "maven_remote_files.sha512",
     maven_remote_files.upstream_checked_at AS "maven_remote_files.upstream_checked_at",
     maven_remote_files.upstream_etag AS "maven_remote_files.upstream_etag"
FROM public.maven_remote_files
     INNER JOIN public.maven_remote_packages ON ((maven_remote_packages.id = maven_remote_files.maven_remote_package_id) AND (maven_remote_packages.namespace_id = maven_remote_files.namespace_id))
WHERE ((((((((maven_remote_files.namespace_id = $1::uuid) AND (maven_remote_packages.namespace_id = $2::uuid)) AND (maven_remote_packages.maven_remote_repository_id = $3::uuid)) AND (maven_remote_packages.group_id = $4::text)) AND (maven_remote_packages.artifact_id = $5::text)) AND (maven_remote_files.file_name = $6::text)) AND (maven_remote_files.maven_remote_version_id IS NULL)) AND (maven_remote_files.soft_deleted_at IS NULL)) AND (maven_remote_packages.soft_deleted_at IS NULL))
LIMIT $7;

Bound args: [<namespace-uuid>, <namespace-uuid>, <maven-remote-repository-uuid>, com.example, myapp, maven-metadata.xml, 1]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Limit  (cost=0.41..16.46 rows=1 width=215) (actual time=0.010..0.011 rows=1 loops=1)
   Buffers: shared hit=5
   ->  Nested Loop  (cost=0.41..16.46 rows=1 width=215) (actual time=0.010..0.010 rows=1 loops=1)
         Join Filter: (maven_remote_packages.id = maven_remote_files.maven_remote_package_id)
         Buffers: shared hit=5
         ->  Index Scan using maven_remote_files_p35_namespace_id_maven_remote_package_id_idx on maven_remote_files_p35 maven_remote_files  (cost=0.12..8.14 rows=1 width=247) (actual time=0.005..0.005 rows=1 loops=1)
               Index Cond: ((namespace_id = '<ns>'::uuid) AND (file_name = 'maven-metadata.xml'::text))
               Buffers: shared hit=2
         ->  Index Scan using maven_remote_packages_p35_namespace_id_maven_remote_reposit_idx on maven_remote_packages_p35 maven_remote_packages  (cost=0.28..8.31 rows=1 width=32) (actual time=0.004..0.004 rows=1 loops=1)
               Index Cond: ((namespace_id = '<ns>'::uuid) AND (maven_remote_repository_id = '<mrr>'::uuid) AND (group_id = 'com.example'::text) AND (artifact_id = 'myapp'::text))
               Buffers: shared hit=3
 Planning:
   Buffers: shared hit=6
 Planning Time: 0.154 ms
 Execution Time: 0.019 ms

Timings: planning 0.154ms, execution 0.019ms, total 0.173ms.

Edited by Moaz Khalifa

Merge request reports