feat(managementapi): maven remote bulk pass (S17 Phase 6 plan: 31/42)

Hayley Swimelarrequested to merge
314/step-31-maven-remote-bulk-pass into main

Why

The Maven bulk delete routes serve hosted repositories only. resolveMavenBulkScope narrows on the hosted child row, so a kind=remote repository answers the existence-hiding 404 and nothing can evict a cached upstream subtree through the bulk surface. The single-artifact eviction arms are a sibling step's. This is the bulk half.

This is Step 31 of the S17 Phase 6 plan. The routes now admit remote repositories and the worker runs the remote pass: subset entries evict as no-op-tolerant sets, and delete_all walks one ordered pass over the collection in its read-route list order under no acceptance-time predicate. The response promises the pass rather than an empty end state, because a live cache can re-fill the moment an entry is evicted.

Spec: docs/specs/S17-rest-management-api.md, AC #108 and AC #102 (closed).

Depends on

The parent of this work merged while it was in review, so half the plan's gate is already satisfied and half is not.

The plan requires the bulk workers' fail-closed kind read to be deployed fleet-wide before any route widens (Approach). The reason is a rolling deploy. The bulk pass crosses River's job queue, where two binaries run at once: a new pod's widened route enqueues a remote batch onto the shared mgmtapi:bulk-delete-maven kind, and an old pod can claim it. Before the fail-closed read, that old pod's worker read a non-hosted kind as a whole-batch no-op and returned nil, so River marked the job completed. A 202'd eviction was lost with no error and no retry. I confirmed that behavior on the pre-merge tree rather than inferring it.

  • Merge half: satisfied. The fail-closed read is on main as c437da899, from fix(managementapi): fail closed on bulk-worker ... (!1736 - merged) • Hayley Swimelar • 19.4.
  • Deploy half: outstanding. docs/dev/releases.md records production promotion as a manual play (staging uploads on every release; the production manifest upload waits on the manual job), so this must not ride the same promotion as that commit's release. Deployment is manifest upload plus Flux reconciliation, so a green pipeline is not evidence the binary runs: verify the fail-closed workers are live via the version field of a recent access log row before merging.

Whether the deploy ordering is required at all is an open operator question, tracked live at Deploy-ordering gate for the remote bulk passes (#755) • Hayley Swimelar • 19.4 (Bulk workers no-op on non-hosted kinds where th... (#723 - closed) • Hayley Swimelar • 19.4 covered the merge half and is closed). While #755 is open this MR merges only after the running fleet is verified or the ordering is relaxed there.

Not a merge gate, but owed before any purger ships. EvictMavenRemotePackage has no production caller on main, so the packages arm here is the first bulk writer of a maven_remote_packages tombstone. LifecycleScanStore carries no scan level for that table, and a package mark writes no other row, so the versions, files, and blob_storage_attachments beneath it stay live and unmarked. Nothing is stranded today, because LifecycleScanStore has no production consumer and no hosted tombstone reaps either. The spec names the state under Eviction markers, and the "S20-A amendment: three eviction scan levels, and the reachable destructive=false branch" bullet in Follow-ups books the maven_remote_packages, npm_remote_packages, and container_remote_images levels. The maven_remote_packages level must land before the purger does. The amendment itself belongs with the S20-A owner.

What

The plan's Files list is short by two packages, and its estimate by 8x. Step 31 names only internal/managementapi/ worker and route files. The worker cannot compile without datastore reads that did not exist, so this MR also adds internal/datastore production code and the composition-root wiring in cmd/artifact-registry/. The plan's size row estimates ~700 total LOC against an actual ~5,833. MavenRemoteBulkScope maps to no plan step at all. Recording this per the plan-contradiction rule. A docs(plans) follow-up should correct the Files list and the estimate.

The remote pass walks each collection in its read-route list order, and the versions direction is load-bearing. Packages sort (group_id, artifact_id) ascending and files sort file_name ascending, but versions sort created_at, id descending, matching the read route's default. The direction is what delivers AC #108's re-cache clause. Under an ascending walk a row re-cached mid-pass is stamped now(), sorts after the cursor, and gets evicted, which is the opposite of what the criterion promises. It also unbounds the pass, so it chases concurrent pulls instead of terminating in ceil(N/100) pages. An earlier revision of this branch had it ascending and the suite stayed green, because the unit fake pages by seeded-slice position and cannot see a direction. The guard is an integration test.

No acceptance-time predicate, and the reason is the race rather than the column. The remote page-params type carries no acceptance instant at all. The predicate the hosted pass applies guards a publish race, and a remote collection takes no publishes: every row is a re-fetch the caller's own traffic caused. The absent created_at column is a secondary reason that does not hold on its own, because maven_remote_versions does record it and it is this cursor's sort key.

Hosted-only couplings stay off the remote arm structurally. mavenBulkOps carries an emitsDeletionEvents flag, false on the remote arm, so AC #102 (closed)'s "evictions emit nothing" is enforced by the arm rather than by an omission a future edit could undo. ops(kind) is the worker's single kind switch and fails closed by default, so a virtual repository still gets the retryable errMavenBulkKindUnarmed.

The marker asymmetry is deliberate. Package and version marks count every success as applied, because the composers do not report an affected-row count. File deletes keep a real applied-or-skipped split, because the composer returns deleted. The pass promises the pass rather than an end state, so an over-count of applied marks costs a log line rather than a wrong answer.

This MR deliberately duplicates two files with an open sibling, and the resolution direction matters. The sibling is feat(managementapi): serve the maven remote pac... (!1726 - merged) • Hayley Swimelar • 19.4.

internal/datastore/maven_remote_eviction_resolve.go here carries three of that MR's four resolve methods. Since 4e0f5bc4c it is no longer a strict subset: two shared symbols diverge. wrapMavenRemoteResolveErr takes (err, entity) here against (err, entity, namespaceID, id) there, and FindMavenRemoteRepositoryID wraps with "finding the maven remote repository binding: %w" here against "finding maven remote repository binding (namespace=%s, repository=%s): %w" there. This copy's identifier-free wraps are the ones docs/dev/database-query-patterns.md requires of a new store method, and neither resolver is old enough for that rule's grandfather clause, so the surviving copy must carry them. Neither direction is mechanical: the three methods this branch calls have byte-identical exported signatures in both copies, so dropping this copy for the superset leaves the build and the tests green with the wraps silently regressed, while the sibling's fourth method calls queryMavenRemoteFilesSelectInRepository, which this branch never declares because it carries no file-level resolve, so taking the superset whole does not compile until that constant comes across with it. The durable fix is on the sibling, mirroring 4e0f5bc4c there.

cmd/artifact-registry/maven_remote_evictor.go is not a subset: the two binding docs name different interface counts, and three query_names.go constants are byte-identical, so "keep both" is a compile error there. Whoever resolves that conflict should keep one copy of each constant and merge the binding docs.

A half-created remote repository answers 202 where its hosted equivalent answers 404. A repository row whose format child row is missing is a broken create invariant, and the remote arm treats it as an empty collection rather than an absence. Deliberate, and noted because it is a visible asymmetry.

One contract-text gap this MR does not fix. api/openapi/v1.yaml defines delete_all completion by an acceptance boundary the remote pass has no equivalent for: line 3311 says a delete_all is "finished once nothing from before the request remains", line 3322 says a resubmission resolves "its own acceptance time", and the three operation descriptions at lines 1061, 1199, and 1345 say delete_all "empties the collection". The spec refuses that end-state promise for remote collections. A caller who builds a poll-until-empty loop from the current text never terminates against a live remote cache. I left the file untouched: the same text is wrong for all three formats' remote arms, so a Maven-only edit leaves the document inconsistent in a new way, and no declared status changes here. This wants one follow-up covering Maven, npm, and container together.

Adjacent and not a deviation: ADR-012 lists repository_kind unnarrowed on artifact_registry_artifact_deleted while narrowing the pulled event to hosted and virtual. The ADR names dimensions and the spec governs emission, so this MR conforms. The handbook amendment is owed separately.

Spec coverage

Spec: docs/specs/S17-rest-management-api.md. AC numbers are the spec's auto-numbered list, extracted rather than counted.

# Criterion (this MR's Maven slice) Tests
AC-108 Bulk eviction accepts the Phase 4 selectors with the same validation on a remote repository TestMavenBulkDeletes_RemoteRepositoryEnqueues202, TestMavenBulkWorker_RemoteKind_ResolvesThroughTheRemoteChain
AC-108 The same no-op rules: absent, already-evicted, foreign, and out-of-scope entries skip rather than fail TestMavenBulkWorker_RemoteSubset_SkipsEntriesOutsideTheChain, ..._RefusesAnotherPackagesVersion, ..._ReMarkIsNotAFailure, TestBulkMavenRemoteWorkerIntegration_CrossRepositoryEntryIsASkip
AC-108 delete_all makes one ordered pass, in list order rather than by id TestMavenBulkWorker_RemoteDeleteAll_WalksTheCollectionsListOrder, the three TestMavenRemoteBulk*PageStmt_SQL tests, the three TestBulkMavenRemoteWorkerIntegration_DeleteAll*_DrainsInListOrder tests
AC-108 Every entry present when the pass reaches its position is evicted, past the page cap TestMavenBulkWorker_RemoteDeleteAll_ExactPageSizeCollection, ..._EmptyCollectionAppliesNothing, TestBulkMavenRemoteWorkerIntegration_DeleteAll_OnAnEmptyCollection
AC-108 An entry re-cached behind the pass survives without failing the job TestBulkMavenRemoteWorkerIntegration_ReCacheOfANewerVersionSurvives (the load-bearing case, shown failing against an ascending walk), ..._ReCacheOfAnEarlierFileSurvives
AC-108 A pass resumes correctly past a cursor row the pass itself evicted TestMavenBulkWorker_RemoteDeleteAll_ResumesPastAnEvictedCursorRow (four cases, every comparison leg mutation-tested)
AC-102 Evictions emit no artifact_registry_artifact_deleted event the tracker-empty assertions on every remote arm, plus the remote rows of the kind-dispatch tables
AC-102 The hosted deletes' emission is unchanged the hosted rows of the kind-dispatch tables, one event each

Error cases

Condition Tests
A kind this binary carries no arm for defers rather than completing TestMavenBulkWorker_KindWithoutAnArm_DefersTheBatch (virtual), retargeted from the pre-arm version with its remote guarantee moved to the resolve test
An unserviceable namespace retries, an absent row no-ops the WriteServiceable verdict rows of the worker tables
A full page with no progress stops before advancing the cursor the drain-stall rows (errMavenBulkDrainStalled)
Tenant isolation: an id under a sibling remote repository or another namespace skips and the row survives TestBulkMavenRemoteWorkerIntegration_CrossRepositoryEntryIsASkip, the chain-leg assertions in the resolver SQL tests
Injection: every statement through the Jet builder with bound parameters the three TestMavenRemoteBulk*PageStmt_SQL tests and the resolver SQL tests, placeholder emission and bind order asserted

Test plan

Every gate below ran on the final rebased tree.

  • gofmt, go build ./..., go vet ./..., and go vet -tags=integration ./...
  • golangci-lint plain on the three touched packages: 0 issues. Integration-tagged with --build-tags=integration --max-same-issues=0 --max-issues-per-linter=0 --uniq-by-line=false: 0 findings in this branch's files, against an unchanged repo-wide baseline that CI lint cannot see.
  • Full unit suite, go test ./... -count=1
  • go test -tags=integration ./internal/managementapi/ ./internal/datastore/ against PostgreSQL 17 tuned with max_locks_per_transaction=4096. The stock setting fails roughly 95 pre-existing tests with SQLSTATE 53200, tracked in Local testcontainers Postgres needs max_locks_p... (#657) • Unassigned.
  • TestEveryStatementIsInstrumented and the TestQueryNames_* set, covering the six new query names.
  • The integration binary was confirmed to boot per rig file rather than inferred from a clean go vet -tags=integration, which compiles those files without ever constructing the handler. All three touched rigs wire the three new required deps, each guarded by a panic in NewMavenBulkWorker, and one real test body from each rig passed.
  • EXPLAIN (ANALYZE, BUFFERS) on all three page reads: index-driven, one partition pruned of 64, the keyset folded into Index Cond with no residual filter. The versions read is an Index Only Scan Backward, so the descending order costs no index of its own.
  • Allocation measured rather than reasoned: the drain allocates 2 objects and roughly 1.9 KB per page and zero per entry, so peak memory is O(page) at about 29 KB whatever the collection size. On a synthetic 1M-entry pass that is 20,222 mallocs across 10,000 pages with a flat live heap. Page-statement construction is 1.7% to 2.1% of a page read, so hoisting it is unwarranted.

No e2e scenario is added, and none is affected. docs/testing/e2e/maven.md scopes virtual and remote Maven repositories out until the capability ships, and the catalog holds no bulk or eviction scenario.

Conformance suites are unaffected. This MR changes management API routes under /api/v1/, not the Maven wire protocol.

The //nolint tokens were measured rather than assumed. Four this branch had added fired nothing and are gone. All six dupl tokens are load-bearing, re-measured after a structural refactor.

Diff size

Roughly 5,833 reviewable LOC (insertions plus deletions against the merge base), past the 500 docs/dev/development-model.md asks a justification for. Test code is 73% of it.

Group LOC Files
Production, internal/ 1,397 5
Composition root, cmd/ 179 2
Tests 4,257 10

Splitting does not help. The only clean cut is a datastore-only precursor at 1,582 LOC, which leaves the remainder at 4,251 and still 8x the ceiling, while separating a test suite from the statements it pins. The other three cuts are foreclosed: the worker arm cannot compile without the datastore types, the route gate cannot ship ahead of the worker arm without opening the same-MR widening window the plan forbids, and the wiring must land with its nil guards or the worker panics at boot.

Database Review Evidence

Queries

Note

This MR adds no migration, so only query mode ran. Plans are from EXPLAIN (ANALYZE, BUFFERS) against an ephemeral PostgreSQL 17.10 container (matching GL_PG_CURR_VERSION from .gitlab-ci-other-versions.yml), with synthesized seed data rolled back per query and the container torn down at the end of the run. Numbers reflect moderate cardinality and do not capture production-scale effects. The Index column names the partitioned parent index. The raw plans name the partition child it resolved to. See Database review evidence for seed sizing, methodology, and the anomalies the skill flags. Expand each row's details for the seed shape, rendered SQL, bound args, and raw plan.

Method Plan node Index Rows (plan / actual) Cost Time Buffers (hit / read) Partitions
datastore.MavenRemoteBulkScope.FileIDsInScope.AfterNil Limit unique_maven_remote_files_ns_id_ver_id_file_name 100 / 100 112.96 0.042ms 9 / 0 1/64
datastore.MavenRemoteBulkScope.FileIDsInScope.AfterSet Limit unique_maven_remote_files_ns_id_ver_id_file_name 100 / 100 199.64 0.040ms 9 / 0 1/64
datastore.MavenRemoteBulkScope.PackageIDsInScope.AfterNil Limit unique_maven_remote_packages_ns_id_repo_id_group_artifact 100 / 100 58.54 0.077ms 76 / 0 1/64
datastore.MavenRemoteBulkScope.PackageIDsInScope.AfterSet Limit unique_maven_remote_packages_ns_id_repo_id_group_artifact 100 / 100 139.46 0.096ms 77 / 0 1/64
datastore.MavenRemoteBulkScope.VersionIDsInScope.AfterNil Limit index_maven_remote_versions_on_ns_id_pkg_id_created_at_id 100 / 100 60.96 0.053ms 7 / 0 1/64
datastore.MavenRemoteBulkScope.VersionIDsInScope.AfterSet Limit index_maven_remote_versions_on_ns_id_pkg_id_created_at_id 100 / 100 197.73 0.089ms 7 / 0 1/64
datastore.MavenRemoteEvictionResolver.FindMavenRemotePackageInRepository Limit pk_maven_remote_packages 1 / 1 8.44 0.043ms 4 / 0 1/64
datastore.MavenRemoteEvictionResolver.FindMavenRemoteRepositoryID Limit unique_maven_remote_repositories_namespace_id_and_repository_id 1 / 1 8.30 0.020ms 3 / 0 1/64
datastore.MavenRemoteEvictionResolver.FindMavenRemoteVersionInRepository Limit unique_maven_remote_versions_id_pkg_id_ns_id, pk_maven_remote_packages 1 / 1 113.14 0.054ms 8 / 0 1/64, 1/64
datastore.MavenRemoteBulkScope.FileIDsInScope.AfterNil

Summary: Plan matches the method's intent. The scan uses unique_maven_remote_files_ns_id_ver_id_file_name, whose leading two columns are the equality legs and whose third is the ORDER BY key, so no Sort node appears and namespace_id prunes to one of 64 partitions. The page stops at 100 of the version's 5000 live files for 9 buffers. No anomalies.

Seed shape: namespaces=1, repositories=1, maven_remote_repositories=1, maven_remote_packages=1, maven_remote_versions=96, blob_storage_blobs=1, blob_storage_attachments=1, maven_remote_files=100000 (5000 under the addressed version, 1000 under each of 95 siblings)

Rendered SQL:

SELECT maven_remote_files.id AS "scope_row.id",
     maven_remote_files.file_name AS "scope_row.file_name"
FROM public.maven_remote_files
WHERE (((maven_remote_files.namespace_id = $1::uuid) AND (maven_remote_files.maven_remote_version_id = $2::uuid)) AND (maven_remote_files.maven_remote_version_id IS NOT NULL)) AND (maven_remote_files.soft_deleted_at IS NULL)
ORDER BY maven_remote_files.file_name ASC
LIMIT $3;

Bound args: ['11111111-1111-4111-8111-111111111111', '82490395-194b-43af-b6bd-ba2330851aa9', 100]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Limit  (cost=0.42..112.96 rows=100 width=44) (actual time=0.011..0.029 rows=100 loops=1)
  Buffers: shared hit=9
  ->  Index Scan using maven_remote_files_p07_namespace_id_maven_remote_version_id_idx on maven_remote_files_p07 maven_remote_files  (cost=0.42..5537.32 rows=4920 width=44) (actual time=0.011..0.024 rows=100 loops=1)
        Index Cond: ((namespace_id = '11111111-1111-4111-8111-111111111111'::uuid) AND (maven_remote_version_id = '82490395-194b-43af-b6bd-ba2330851aa9'::uuid))
        Buffers: shared hit=9
Planning:
  Buffers: shared hit=232 read=2
Planning Time: 1.142 ms
Execution Time: 0.042 ms

Timings: planning 1.142ms, execution 0.042ms, total 1.184ms.

datastore.MavenRemoteBulkScope.FileIDsInScope.AfterSet

Summary: Same index as the first-page arm, with file_name > $3 folded into the Index Cond rather than re-checked as a Filter, which is what makes a resumed page a seek instead of a scan and discard. Buffers hold at 9 against the first page's 9, so resuming from the middle of the collection costs what starting it does. No anomalies.

Seed shape: namespaces=1, repositories=1, maven_remote_repositories=1, maven_remote_packages=1, maven_remote_versions=96, blob_storage_blobs=1, blob_storage_attachments=1, maven_remote_files=100000 (5000 under the addressed version, 1000 under each of 95 siblings)

Rendered SQL:

SELECT maven_remote_files.id AS "scope_row.id",
     maven_remote_files.file_name AS "scope_row.file_name"
FROM public.maven_remote_files
WHERE ((((maven_remote_files.namespace_id = $1::uuid) AND (maven_remote_files.maven_remote_version_id = $2::uuid)) AND (maven_remote_files.maven_remote_version_id IS NOT NULL)) AND (maven_remote_files.soft_deleted_at IS NULL)) AND (maven_remote_files.file_name > $3::text)
ORDER BY maven_remote_files.file_name ASC
LIMIT $4;

Bound args: ['11111111-1111-4111-8111-111111111111', '5b9b932d-f616-4641-94b5-fe760c32fa7c', 'review-prep-file-002501.jar', 100]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Limit  (cost=0.42..199.64 rows=100 width=44) (actual time=0.012..0.030 rows=100 loops=1)
  Buffers: shared hit=9
  ->  Index Scan using maven_remote_files_p07_namespace_id_maven_remote_version_id_idx on maven_remote_files_p07 maven_remote_files  (cost=0.42..263.39 rows=132 width=44) (actual time=0.011..0.025 rows=100 loops=1)
        Index Cond: ((namespace_id = '11111111-1111-4111-8111-111111111111'::uuid) AND (maven_remote_version_id = '5b9b932d-f616-4641-94b5-fe760c32fa7c'::uuid) AND (file_name > 'review-prep-file-002501.jar'::text))
        Buffers: shared hit=9
Planning:
  Buffers: shared hit=7 read=7 written=7
Planning Time: 0.324 ms
Execution Time: 0.040 ms

Timings: planning 0.324ms, execution 0.040ms, total 0.364ms.

datastore.MavenRemoteBulkScope.PackageIDsInScope.AfterNil

Summary: Index Scan over unique_maven_remote_packages_ns_id_repo_id_group_artifact, the partial unique index that covers both equality legs and both ORDER BY columns, so the walk needs no Sort. One of 64 partitions, 100 rows out of the remote repository's 5000. No anomalies.

Seed shape: namespaces=1, repositories=96, maven_remote_repositories=96, maven_remote_packages=100000 (5000 under the addressed remote repository, 1000 under each of 95 siblings)

Rendered SQL:

SELECT maven_remote_packages.id AS "scope_row.id",
     maven_remote_packages.group_id AS "scope_row.group_id",
     maven_remote_packages.artifact_id AS "scope_row.artifact_id"
FROM public.maven_remote_packages
WHERE ((maven_remote_packages.namespace_id = $1::uuid) AND (maven_remote_packages.maven_remote_repository_id = $2::uuid)) AND (maven_remote_packages.soft_deleted_at IS NULL)
ORDER BY maven_remote_packages.group_id ASC, maven_remote_packages.artifact_id ASC
LIMIT $3;

Bound args: ['11111111-1111-4111-8111-111111111111', '138f1b01-c0f1-4f66-bd83-c2468f66833d', 100]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Limit  (cost=0.42..58.54 rows=100 width=49) (actual time=0.012..0.063 rows=100 loops=1)
  Buffers: shared hit=76
  ->  Index Scan using maven_remote_packages_p07_namespace_id_maven_remote_reposit_idx on maven_remote_packages_p07 maven_remote_packages  (cost=0.42..2970.68 rows=5110 width=49) (actual time=0.011..0.058 rows=100 loops=1)
        Index Cond: ((namespace_id = '11111111-1111-4111-8111-111111111111'::uuid) AND (maven_remote_repository_id = '138f1b01-c0f1-4f66-bd83-c2468f66833d'::uuid))
        Buffers: shared hit=76
Planning:
  Buffers: shared hit=158
Planning Time: 0.674 ms
Execution Time: 0.077 ms

Timings: planning 0.674ms, execution 0.077ms, total 0.751ms.

datastore.MavenRemoteBulkScope.PackageIDsInScope.AfterSet

Summary: This is the plan that checks mavenRemoteBulkPackagePageStmt's pg.WRAP claim. The row-value bound renders as ROW(group_id, artifact_id) > ROW($3, $4) inside the Index Cond, not as a post-scan Filter, so Postgres folds it into the index range bound as the comment says. 77 buffers against the first page's 76. No anomalies.

Seed shape: namespaces=1, repositories=96, maven_remote_repositories=96, maven_remote_packages=100000 (5000 under the addressed remote repository, 1000 under each of 95 siblings)

Rendered SQL:

SELECT maven_remote_packages.id AS "scope_row.id",
     maven_remote_packages.group_id AS "scope_row.group_id",
     maven_remote_packages.artifact_id AS "scope_row.artifact_id"
FROM public.maven_remote_packages
WHERE (((maven_remote_packages.namespace_id = $1::uuid) AND (maven_remote_packages.maven_remote_repository_id = $2::uuid)) AND (maven_remote_packages.soft_deleted_at IS NULL)) AND ((maven_remote_packages.group_id, maven_remote_packages.artifact_id) > ($3::text, $4::text))
ORDER BY maven_remote_packages.group_id ASC, maven_remote_packages.artifact_id ASC
LIMIT $5;

Bound args: ['11111111-1111-4111-8111-111111111111', '40d4bad4-8c41-4eb2-91be-91d9db29b934', 'com.example.g025', 'artifact-000025', 100]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Limit  (cost=0.42..139.46 rows=100 width=49) (actual time=0.017..0.067 rows=100 loops=1)
  Buffers: shared hit=77
  ->  Index Scan using maven_remote_packages_p07_namespace_id_maven_remote_reposit_idx on maven_remote_packages_p07 maven_remote_packages  (cost=0.42..3419.37 rows=2459 width=49) (actual time=0.016..0.062 rows=100 loops=1)
        Index Cond: ((namespace_id = '11111111-1111-4111-8111-111111111111'::uuid) AND (maven_remote_repository_id = '40d4bad4-8c41-4eb2-91be-91d9db29b934'::uuid) AND (ROW(group_id, artifact_id) > ROW('com.example.g025'::text, 'artifact-000025'::text)))
        Buffers: shared hit=77
Planning:
  Buffers: shared hit=4
Planning Time: 0.138 ms
Execution Time: 0.096 ms

Timings: planning 0.138ms, execution 0.096ms, total 0.234ms.

datastore.MavenRemoteBulkScope.VersionIDsInScope.AfterNil

Summary: Index Only Scan Backward over index_maven_remote_versions_on_ns_id_pkg_id_created_at_id, so the newest-first order costs no Sort and no index of its own, which is what the builder's comment claims. The 100 heap fetches come from seeding inside the rolled-back transaction, where the visibility map is never set. Against a committed table the page is served from the index alone. No anomalies.

Seed shape: namespaces=1, repositories=1, maven_remote_repositories=1, maven_remote_packages=96, maven_remote_versions=100000 (5000 under the addressed package, 1000 under each of 95 siblings)

Rendered SQL:

SELECT maven_remote_versions.id AS "scope_row.id",
     maven_remote_versions.created_at AS "scope_row.created_at"
FROM public.maven_remote_versions
WHERE ((maven_remote_versions.namespace_id = $1::uuid) AND (maven_remote_versions.maven_remote_package_id = $2::uuid)) AND (maven_remote_versions.soft_deleted_at IS NULL)
ORDER BY maven_remote_versions.created_at DESC, maven_remote_versions.id DESC
LIMIT $3;

Bound args: ['11111111-1111-4111-8111-111111111111', 'cee7b5f1-2cab-4aaf-a424-ee0f0a08881e', 100]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Limit  (cost=0.42..60.96 rows=100 width=24) (actual time=0.022..0.039 rows=100 loops=1)
  Buffers: shared hit=7
  ->  Index Only Scan Backward using maven_remote_versions_p07_namespace_id_maven_remote_packag_idx4 on maven_remote_versions_p07 maven_remote_versions  (cost=0.42..2962.53 rows=4893 width=24) (actual time=0.021..0.034 rows=100 loops=1)
        Index Cond: ((namespace_id = '11111111-1111-4111-8111-111111111111'::uuid) AND (maven_remote_package_id = 'cee7b5f1-2cab-4aaf-a424-ee0f0a08881e'::uuid))
        Heap Fetches: 100
        Buffers: shared hit=7
Planning:
  Buffers: shared hit=289 read=2
Planning Time: 1.272 ms
Execution Time: 0.053 ms

Timings: planning 1.272ms, execution 0.053ms, total 1.325ms.

datastore.MavenRemoteBulkScope.VersionIDsInScope.AfterSet

Summary: Same backward index-only scan, with the descending row-value bound in the Index Cond. This arm is the one sensitive to seed shape: put every version in the namespace under a single package and maven_remote_package_id stops discriminating, at which point the planner prefers the narrower index_maven_remote_versions_on_ns_id_created_at plus an Incremental Sort. That variant measured 0.056 ms, so it is a cost tie rather than a risk, and it is why this seed carries 95 sibling packages. No anomalies.

Seed shape: namespaces=1, repositories=1, maven_remote_repositories=1, maven_remote_packages=96, maven_remote_versions=100000 (5000 under the addressed package, 1000 under each of 95 siblings)

Rendered SQL:

SELECT maven_remote_versions.id AS "scope_row.id",
     maven_remote_versions.created_at AS "scope_row.created_at"
FROM public.maven_remote_versions
WHERE (((maven_remote_versions.namespace_id = $1::uuid) AND (maven_remote_versions.maven_remote_package_id = $2::uuid)) AND (maven_remote_versions.soft_deleted_at IS NULL)) AND ((maven_remote_versions.created_at, maven_remote_versions.id) < ($3::timestamp with time zone, $4::uuid))
ORDER BY maven_remote_versions.created_at DESC, maven_remote_versions.id DESC
LIMIT $5;

Bound args: ['11111111-1111-4111-8111-111111111111', 'c38093d4-816e-4011-8d78-b210b7a8bd15', '2026-08-20 16:13:36.334805+00', '27c17feb-a320-4435-9189-156d9b93f078', 100]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Limit  (cost=0.42..197.73 rows=100 width=24) (actual time=0.031..0.050 rows=100 loops=1)
  Buffers: shared hit=7
  ->  Index Only Scan Backward using maven_remote_versions_p07_namespace_id_maven_remote_packag_idx4 on maven_remote_versions_p07 maven_remote_versions  (cost=0.42..235.22 rows=119 width=24) (actual time=0.030..0.045 rows=100 loops=1)
        Index Cond: ((namespace_id = '11111111-1111-4111-8111-111111111111'::uuid) AND (maven_remote_package_id = 'c38093d4-816e-4011-8d78-b210b7a8bd15'::uuid) AND (ROW(created_at, id) < ROW('2026-08-20 16:13:36.334805+00'::timestamp with time zone, '27c17feb-a320-4435-9189-156d9b93f078'::uuid)))
        Heap Fetches: 100
        Buffers: shared hit=7
Planning:
  Buffers: shared hit=13
Planning Time: 0.234 ms
Execution Time: 0.089 ms

Timings: planning 0.234ms, execution 0.089ms, total 0.323ms.

datastore.MavenRemoteEvictionResolver.FindMavenRemotePackageInRepository

Summary: Single-row seek on pk_maven_remote_packages. The id and namespace address the row and the repository and soft-delete legs land as a Filter, which is the addressing and narrowing split the method's doc comment describes. 4 buffers against 100000 seeded packages, one partition. No anomalies.

Seed shape: namespaces=1, repositories=96, maven_remote_repositories=96, maven_remote_packages=100000 (5000 under the addressed remote repository, 1000 under each of 95 siblings)

Rendered SQL:

SELECT maven_remote_packages.namespace_id AS "maven_remote_packages.namespace_id",
     maven_remote_packages.id AS "maven_remote_packages.id",
     maven_remote_packages.maven_remote_repository_id AS "maven_remote_packages.maven_remote_repository_id",
     maven_remote_packages.group_id AS "maven_remote_packages.group_id",
     maven_remote_packages.artifact_id AS "maven_remote_packages.artifact_id",
     maven_remote_packages.last_downloaded_at AS "maven_remote_packages.last_downloaded_at",
     maven_remote_packages.soft_deleted_at AS "maven_remote_packages.soft_deleted_at"
FROM public.maven_remote_packages
WHERE (((maven_remote_packages.namespace_id = $1::uuid) AND (maven_remote_packages.id = $2::uuid)) AND (maven_remote_packages.maven_remote_repository_id = $3::uuid)) AND (maven_remote_packages.soft_deleted_at IS NULL)
LIMIT $4;

Bound args: ['11111111-1111-4111-8111-111111111111', '66564bfb-813f-4e88-8100-7fe0664a387d', '96e1b18f-b540-4879-9f50-c0e334673376', 1]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Limit  (cost=0.42..8.44 rows=1 width=97) (actual time=0.024..0.025 rows=1 loops=1)
  Buffers: shared hit=4
  ->  Index Scan using maven_remote_packages_p07_pkey on maven_remote_packages_p07 maven_remote_packages  (cost=0.42..8.44 rows=1 width=97) (actual time=0.023..0.023 rows=1 loops=1)
        Index Cond: ((id = '66564bfb-813f-4e88-8100-7fe0664a387d'::uuid) AND (namespace_id = '11111111-1111-4111-8111-111111111111'::uuid))
        Filter: ((soft_deleted_at IS NULL) AND (maven_remote_repository_id = '96e1b18f-b540-4879-9f50-c0e334673376'::uuid))
        Buffers: shared hit=4
Planning:
  Buffers: shared hit=7
Planning Time: 0.304 ms
Execution Time: 0.043 ms

Timings: planning 0.304ms, execution 0.043ms, total 0.347ms.

datastore.MavenRemoteEvictionResolver.FindMavenRemoteRepositoryID

Summary: Index Scan over the child table's unique_maven_remote_repositories_namespace_id_and_repository_id with both key columns bound as equalities and namespace_id pruning to one of 64 partitions, exactly what findMavenRemoteRepositoryIDStmt's comment claims. 3 buffers against 5000 seeded child rows. No anomalies.

Seed shape: namespaces=1, repositories=5000, maven_remote_repositories=5000

Rendered SQL:

SELECT maven_remote_repositories.id AS "maven_remote_repositories.id"
FROM public.maven_remote_repositories
WHERE (maven_remote_repositories.namespace_id = $1::uuid) AND (maven_remote_repositories.repository_id = $2::uuid)
LIMIT $3;

Bound args: ['11111111-1111-4111-8111-111111111111', '8aa4d0dc-e557-4043-b9b6-1507c465a9b5', 1]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Limit  (cost=0.28..8.30 rows=1 width=16) (actual time=0.007..0.007 rows=1 loops=1)
  Buffers: shared hit=3
  ->  Index Scan using maven_remote_repositories_p07_namespace_id_repository_id_idx on maven_remote_repositories_p07 maven_remote_repositories  (cost=0.28..8.30 rows=1 width=16) (actual time=0.007..0.007 rows=1 loops=1)
        Index Cond: ((namespace_id = '11111111-1111-4111-8111-111111111111'::uuid) AND (repository_id = '8aa4d0dc-e557-4043-b9b6-1507c465a9b5'::uuid))
        Buffers: shared hit=3
Planning:
  Buffers: shared hit=89
Planning Time: 0.390 ms
Execution Time: 0.020 ms

Timings: planning 0.390ms, execution 0.020ms, total 0.41ms.

datastore.MavenRemoteEvictionResolver.FindMavenRemoteVersionInRepository

Summary: Nested Loop of two single-row index seeks: the leaf on unique_maven_remote_versions_id_pkg_id_ns_id, then its parent package on pk_maven_remote_packages, with both soft-delete legs and the repository terminator applied as Filters. One partition of each table and 8 buffers total, so the chain check costs one extra page read over the package probe. No anomalies.

Seed shape: namespaces=1, repositories=1, maven_remote_repositories=1, maven_remote_packages=96, maven_remote_versions=100000 (5000 under the addressed package, 1000 under each of 95 siblings)

Rendered SQL:

SELECT maven_remote_versions.namespace_id AS "maven_remote_versions.namespace_id",
     maven_remote_versions.id AS "maven_remote_versions.id",
     maven_remote_versions.maven_remote_package_id AS "maven_remote_versions.maven_remote_package_id",
     maven_remote_versions.version AS "maven_remote_versions.version",
     maven_remote_versions.size_bytes AS "maven_remote_versions.size_bytes",
     maven_remote_versions.created_at AS "maven_remote_versions.created_at",
     maven_remote_versions.last_downloaded_at AS "maven_remote_versions.last_downloaded_at",
     maven_remote_versions.soft_deleted_at AS "maven_remote_versions.soft_deleted_at"
FROM public.maven_remote_versions
     INNER JOIN public.maven_remote_packages ON ((maven_remote_packages.namespace_id = maven_remote_versions.namespace_id) AND (maven_remote_packages.id = maven_remote_versions.maven_remote_package_id))
WHERE ((((maven_remote_versions.namespace_id = $1::uuid) AND (maven_remote_versions.id = $2::uuid)) AND (maven_remote_versions.soft_deleted_at IS NULL)) AND (maven_remote_packages.soft_deleted_at IS NULL)) AND (maven_remote_packages.maven_remote_repository_id = $3::uuid)
LIMIT $4;

Bound args: ['11111111-1111-4111-8111-111111111111', '812f04ed-aee9-4981-9432-a93bf032345d', 'f5d76dd1-cdde-40b7-b0a7-ea734780aa0f', 1]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Limit  (cost=0.81..113.14 rows=1 width=87) (actual time=0.027..0.027 rows=1 loops=1)
  Buffers: shared hit=8
  ->  Nested Loop  (cost=0.81..113.14 rows=1 width=87) (actual time=0.026..0.027 rows=1 loops=1)
        Buffers: shared hit=8
        ->  Index Scan using maven_remote_versions_p07_id_maven_remote_package_id_namesp_idx on maven_remote_versions_p07 maven_remote_versions  (cost=0.42..8.44 rows=1 width=87) (actual time=0.013..0.013 rows=1 loops=1)
              Index Cond: ((id = '812f04ed-aee9-4981-9432-a93bf032345d'::uuid) AND (namespace_id = '11111111-1111-4111-8111-111111111111'::uuid))
              Filter: (soft_deleted_at IS NULL)
              Buffers: shared hit=4
        ->  Index Scan using maven_remote_packages_p07_pkey on maven_remote_packages_p07 maven_remote_packages  (cost=0.39..104.69 rows=1 width=32) (actual time=0.012..0.012 rows=1 loops=1)
              Index Cond: ((id = maven_remote_versions.maven_remote_package_id) AND (namespace_id = '11111111-1111-4111-8111-111111111111'::uuid))
              Filter: ((soft_deleted_at IS NULL) AND (maven_remote_repository_id = 'f5d76dd1-cdde-40b7-b0a7-ea734780aa0f'::uuid))
              Buffers: shared hit=4
Planning:
  Buffers: shared hit=87 read=3
Planning Time: 0.949 ms
Execution Time: 0.054 ms

Timings: planning 0.949ms, execution 0.054ms, total 1.003ms.

Related to #314

Edited by Hayley Swimelar

Merge request reports