feat(managementapi): emit accounting deltas on remote cache eviction

Pawel Rozlachrequested to merge
prozlach/issue-775-npm-maven-remote-eviction-deltas into main

What this delivers

The npm and Maven remote cache eviction arms now report the counter deltas their own writes move.

Before this change the remote arms moved rows and reported nothing. So repositories.artifacts_count and repositories.size_bytes read too high until a reconciliation pass corrected them.

What each arm emits now:

  • A package mark emits -N on artifacts_count, where N is the count of live cached versions the mark hid.
  • A version mark emits -1 on artifacts_count.
  • A file delete emits -S on size_bytes, and only when it removed the repository's last reference to that blob.
  • No remote eviction emits a namespace delta. namespace_statistics.components_count settles at the reap.

The cause

The defect was not the emit site. It was one layer below, in the return shape of the six eviction composers.

Those composers live in internal/datastore/maven_remote_eviction.go and internal/datastore/npm_remote_eviction.go. Each issued its write and then discarded the values a counter delta needs. Both marks per format threw away the affected-row count of their UPDATE and returned an error alone. The two file deletes reported a removal without the bytes it freed.

The Maven bulk arm already reached emitDeleteCounters and passed amounts that the helper's own zero-suppression then dropped. The npm remote arm reached no emit at all.

Two new statements support the file arm. They sit in internal/datastore/repo_blob_references.go, beside the hosted probes that file already holds. Each is a last-reference probe, one per format. The npm probe walks npm_remote_files and npm_remote_metadata_files, which is the row set recomputeNpmRemoteFilesSizeStmt walks. Neither probe filters soft_deleted_at, because neither recompute does.

Behavior change beyond the accounting

Four things change here that no counter reads.

  • A re-marked entry now counts as skipped, and the two pass-log keys are shared. Both bulk arms gate the emit on the applied value the composer returns. A re-marked entry therefore lands in skipped, where it landed in applied before. Those two key spellings belong to the container bulk pass, at internal/managementapi/bulk_container_worker.go:82-83. The npm remote pass reuses them at internal/managementapi/bulk_npm_worker_remote.go:704-705. The Maven pass writes the same two strings at internal/managementapi/bulk_maven_worker.go:888-889. An operator query that already counts on those keys reads differently after this deploys.
  • A failed accounting read now rolls back an eviction that would have committed. The file-delete composer returns fileFreedSizeTx's error rather than dropping it. See internal/datastore/maven_remote_eviction.go:278-281 and internal/datastore/npm_remote_eviction.go:322-325. That probe runs inside the delete's own transaction. Its error therefore rolls back the row deletion as well.
  • The package mark holds the package row's write lock across the count. The mark's UPDATE takes the row lock, and the live-version count runs before the commit releases it. The pull path writes that same row. It updates last_downloaded_at at internal/datastore/maven_remote_cache.go:302 and :698, and at internal/datastore/npm_remote_packages.go:385. A download of the package under eviction therefore waits for the count.
  • The six single-target routes pay for work they cannot use until part 2. The two package arms open a transaction and add a COUNT(*), where each ran one bare UPDATE before. The two file arms keep the transaction they had and add a repository resolve and an EXISTS probe. A size read follows on the arm that frees the blob. The two version arms cost nothing new, because the row count is read from the command tag they already had. All six discard what the composer returns, so main carries this cost and part 2 brings the benefit.

This is part 1 of two

Part 1 widens the composers, the two probes, and the two bulk delete workers.

The six single-target remote DELETE routes still emit no counter at all:

  • internal/managementapi/package_delete.go:202 (Maven) and :330 (npm)
  • internal/managementapi/version_delete.go:236 (Maven) and :355 (npm)
  • internal/managementapi/file_delete.go:312 (Maven) and :492 (npm)

Each of those call sites takes the new return values and discards them with _. A run against the live service confirmed it. All six routes answered 202, the target rows changed, and no counter moved after two drain intervals.

Part 2 adds the emit at those six routes, with their tests. Part 2 cannot land first, because the composers report no amount until part 1 lands.

This branch also removes the last in-code pointer to the issue. git grep -- '#775' -- '*.go' returns nothing at the tip, against internal/managementapi/bulk_maven_worker.go:595 at the merge base. This section is therefore the only remaining pointer to the work part 2 owes.

Test coverage

Source: the issue states no acceptance criteria of its own. These items were derived from the issue body and from docs/specs/S22-storage-accounting.md, then narrowed to the bulk arms and the composers. Item 16 is this branch's own.

# Item Tests
1 A remote package eviction that applied emits one repository delta (-N, 0), both formats TestMavenBulkWorker_RemoteArm_EmitsTheDeltasItsTransactionsMoved/{subset,delete_all}_packages…, TestBulkDeleteNpmWorker_RemoteArm_EmitsTheDeltasItsTransactionsMoved/{subset,delete_all}_packages…, Test{Maven,Npm}RemoteEvictor_Evict*RemotePackage_ReportsWhatTheMarkHid, which now also pins a sibling cached package and a package with no live version
2 A remote version eviction that applied emits one repository delta (-1, 0), both formats the same two tables' …versions carry one each cases, Test{Maven,Npm}RemoteEvictor_Evict*RemoteVersion_ReportsWhetherTheMarkApplied
3 A remote file eviction that removed a row emits (0, -S), and only on the repository's last reference to the blob the same two tables' …files carry the freed bytes cases, Test{Maven,Npm}RemoteEvictor_Delete*RemoteFile_ReportsTheBytesItFreed/the repository's last reference frees the blob's size
4 A still-referenced blob emits nothing, not a zero-valued delta …/a still-referenced blob emits nothing (Maven), …/a blob a cached packument still holds emits nothing (npm); TestDeleteCountersNeedsDispatchMatchesEmit unchanged and green
5 No remote eviction at any level emits a namespace delta every case of both unit tables asserts namespaceEmits() empty, as do both recompute-agreement integration tests
6 A mark emits no size delta; a file eviction emits no artifact delta the exact fakeRepoCounterEmit equality in both unit tables
7 An idempotent re-mark, and a file eviction that found the row gone, emit nothing …/a re-marked cached {package,version} emits nothing and …/a cached file already gone emits nothing in both unit tables; …/a repeat mark reports no write and no count and …/a delete that removed nothing frees nothing in both datastore suites
8 The package mark's hidden-version count is read inside the mark's own transaction Test{Maven,Npm}RemoteEvictor_Evict*RemotePackage_ReportsWhatTheMarkHid/the transaction's report survives the reap a recount would not, TestMavenRemoteEvictor_EvictionMarks_SurviveTheCallersRollback/a package mark survives the caller's rollback; partial, see note
9 The Maven probe walks maven_remote_files joined to the repository through maven_remote_packages TestMavenRemoteEvictor_DeleteMavenRemoteFile_ReportsTheBytesItFreed/a package-level row holding the digest frees nothing; the Maven recompute-agreement test seeds a package-level maven-metadata.xml
10 The npm probe walks npm_remote_files UNION npm_remote_metadata_files TestNpmRemoteEvictor_DeleteNpmRemoteFile_ReportsTheBytesItFreed/a cached packument holding the digest frees nothing; the npm recompute-agreement test seeds a packument
11 Neither probe filters soft_deleted_at Maven …/a row under a marked cache package holding the digest frees nothing; npm …/a sibling under an evicted version holding the digest frees nothing; …/a tombstoned sibling row holding the digest frees nothing in both formats
12 The emitted deltas equal the change the recompute sees TestBulkMavenRemoteWorkerIntegration_EmittedDeltasAgreeWithTheRecompute, TestIntegration_BulkDeleteNpmWorker_RemoteEmittedDeltasAgreeWithTheRecompute
13 namespace_statistics.components_count is unchanged by every remote eviction the same namespaceEmits() assertions as item 5
14 The container remote arm and every hosted arm emit exactly what they emit today hosted: TestMavenBulkWorker_HostedArm_StillEmits, TestBulkDeleteNpmWorker_HostedArm_StillEmits, and the three *_EmitsPerAppliedEntry suites, all green. Container remote: no new test, see note
15 The six new statement names stay inside the query-name cardinality budget No test covers this, because !1948 (merged) removed internal/metrics/name_budget_test.go from origin/main. The count is by hand: 463 declared names against a budget of 500
16 The six single-target route arms still emit nothing, and no test asserts that they do held by the diff: signature-only hunks, and artifact_delete_counters_test.go still carries no remote arm

Two declared coverage gaps.

The gap in item 8 is narrower than it was, and it is still open. Both formats now carry …/the transaction's report survives the reap a recount would not. That case pins that the report and a recount taken after the reap give different numbers. It does not pin where the count was taken. The composer commits before it returns, so the point at which the count ran is not observable from outside. An implementation that recounted straight after the commit passes the same case. No seam in this change can carry the stronger claim. The hosted precedent MavenBulkMarkers.MarkMavenPackage has no test of this shape either.

Item 14's container half has no new test, because nothing here reaches the container arm. A positive "emits nothing" pin there is one that issue #837 (closed), the container remote eviction, has to invert.

One coverage loss, stated plainly, because the claim splits between the two marks. This change deletes two unit tests: TestMavenRemoteEvictor_EvictMavenRemotePackage_ExecError and TestMavenRemoteEvictor_EvictMavenRemoteVersion_ExecError.

For the version mark the coverage moved to the integration tier. EvictMavenRemoteVersion still reaches instrumentExec directly at internal/datastore/maven_remote_eviction.go:127. TestMavenRemoteEvictor_EvictionMarks_CancelledContextWritesNothing asserts its evicting maven remote version wrap. Every guard returns its sentinel bare, so only a path that ran the UPDATE can carry that text.

For the package mark the coverage is gone, not moved. EvictMavenRemotePackage now opens a transaction, and database/sql returns ctx.Err() before any driver work. BeginTx therefore fails first, under the same outer evicting maven remote package wrap a driver error would carry. The inner marking the package: %w wrap at internal/datastore/maven_remote_eviction.go:205 is the only text that separates the two, and nothing in the tree asserts it. No remedy is in reach either. A settled decision removed the db qrm.DB parameter that let a unit test inject a driver error. No other seam can make that UPDATE fail inside a live transaction.

One note on item 15. origin/main no longer carries internal/metrics/name_budget_test.go. !1948 (merged) merged and removed it, so nothing in CI counts the names now. The hand count and its two inputs are under ## Merge order.

End-to-end scenario catalogs

docs/testing/ was updated, so this merge request owes no "why not".

Three lines change, across the two catalogs. Two of them are the …lifecycle.management-delete-storage-counters row of each catalog's ## Lifecycle table. That row already owns every storage-counter claim for its format. The third is the same scenario's row in maven's ## Usage data table.

  • docs/testing/e2e/maven.md:136 gains the remote sentence. artifacts_count falls by the count of live cached versions a package mark hid, and by one per version mark. size_bytes falls by a cached file's blob size exactly when the delete took the repository's last reference to that blob.
  • docs/testing/e2e/npm.md:187 gains the same sentence. It replaces a claim this branch falsifies: "Against a remote repository it evicts the cached rows and moves no counter at all".
  • docs/testing/e2e/maven.md:191 scopes its event promise to hosted deletes. The row promised one artifact_registry_artifact_deleted per committed delete the journey performs. This branch puts remote bulk deletes inside that journey, and the remote arm emits no event.

Both ## Lifecycle rows also now state that the single-target remote DELETE routes move no counter. Every added counter claim was confirmed against the running service before it was written. The event claim is read off the code instead: internal/managementapi/bulk_maven_worker.go:644 sets emitsDeletionEvents: false for the remote arm.

No new row was added. This branch adds no route, no status, and no scenario. It changes the numeric outcome of a scenario both catalogs already carry, and that outcome is what the row states.

docs/dev/storage-accounting.md is deliberately not updated here. A reference page claiming that the remote path emits its deltas is false for six of twelve call sites until part 2 merges. That page belongs to part 2.

Size

The diff is past 500 reviewable lines, so here is the split.

Measured at the branch tip bb5483ec9 with git diff --shortstat origin/main..bb5483ec9, where origin/main is f8efbca1c: 39 files, 2392 insertions, 1006 deletions. The per-group figures come from git diff --numstat over the same range. The production subtotal covers the four Go groups above it. The two catalog files are Markdown and sit outside that subtotal.

Group Files Insertions Deletions
Datastore composers and probes 4 371 271
Management-API bulk arms 3 105 119
Single-target route seams, signature only 3 21 18
Composition root 2 24 41
Production subtotal 12 521 449
e2e scenario catalogs 2 3 3
Tests 25 1868 554
Total 39 2392 1006

The work is already split, and this is part 1 of two. Part 2 takes the six single-target route arms and their tests, and this merge request does not carry them. So the answer to "split or justify" is that the split happened. The rest of this section is why the remainder does not divide again.

Nearly four fifths of the diff is tests. 1868 of the 2392 inserted lines are test code. The reviewable production surface is 521 lines across 12 files. 371 of those sit in four datastore files, and 342 of the 371 sit in the two eviction files alone. Those two carry the same statement families once per format, so a reviewer reads that shape twice.

Splitting the tests from the implementation is not allowed. The --no-verify carve-out for a test-first commit carries a same-branch clause. The branch that lands a test the code does not yet satisfy must also add the fix. So the test lines and the implementation lines travel together.

Splitting the implementation by format ships a broken half. One mavenRemoteEvictorBinding in cmd/artifact-registry/maven_remote_evictor.go satisfies all four Maven evictor interfaces, and a Go type carries one method per name. The three npm evictor interfaces are declared once in internal/managementapi/bulk_npm_worker_remote.go and typed into both the bulk deps and the handler deps. A composer signature change therefore reaches every consumer in the same compile.

Splitting the two new probes from their callers ships dead statements. mavenRemoteRepoStillReferencesBlobStmt and npmRemoteRepoStillReferencesBlobStmt have one caller each, and both callers are the remote file-eviction arms in this same diff.

The three route files are signature-only. package_delete.go, version_delete.go and file_delete.go contribute 21 insertions and 18 deletions between them, and no emit. They appear for the same shared-seam reason as above.

Reading order for a reviewer:

  1. internal/datastore/query_names.go — the six new statement names.
  2. internal/datastore/repo_blob_references.go — the two remote last-reference probes this branch adds beside the hosted ones.
  3. The two datastore eviction files — the composers, one per format, the same shape twice.
  4. The three management-API bulk files — the emit sites that consume what the composers now report.
  5. The two catalog rows and the test suites.

ADR conformance

ADR-002, ADR-009, ADR-010, ADR-011 and ADR-025 raise no conflict with this change. ADR-007 needs four points, because this branch deviates from one of its merged clauses.

  1. The branch follows the merged spec. docs/specs/S22-storage-accounting.md:174 makes an emit legitimate exactly where the operation changes what its counter's recompute returns, with the delta equal to that change. Its "Emit-site attribution" section fires the size delta when the row and its attachment link actually go. The remote file delete removes both inside its own transaction.
  2. ADR-007 as merged says something else. docs/adr/007_database_schema.md:2101 still places the size_bytes decrement at garbage collection. This branch deviates from that clause as merged.
  3. An amendment is open, and it is evidence rather than authority. Handbook merge request !20887 moves the decrement to the delete. It is unmerged, so it settles nothing yet.
  4. A widening is owed even after that amendment merges. Its replacement wording enumerates a format's own delete and the lifecycle purger. A management-API remote cache eviction is neither. The same gap already sits on main, at the merged management-API hosted file delete: internal/managementapi/artifact_write.go:280, the deleteCounterTargetFile emit arm, decrements size_bytes inside the request.

Part 1 is not gated on that amendment.

Merge order

internal/metrics/cardinality.go — the hunk is already dropped. The branch raised labelName from 450 to 500, and origin/main at f8efbca1c already reads labelName: 500. The rebase onto f8efbca1c took that file from origin/main, so internal/metrics/cardinality.go is no longer in this diff and no conflict is left to resolve. Open merge request !1916 (merged) still carries the same redundant hunk. !1935 (merged) edits the same file, but not that line. !1948 (merged) merged and removed internal/metrics/name_budget_test.go, so no test counts the declared names now. The count is therefore by hand, at the tip bb5483ec9. internal/datastore/query_names.go declares 451 and internal/storage/queries.go declares 12, so this branch declares 463. That figure sits under the budget of 500, and !1916 (merged) adds 9 more. origin/main moves, so re-derive the figure after any further rebase rather than reusing 463.

!1916 (merged) — complementary, and it must not double-emit with this change. It adds internal/lifecycle/emit.go, which emits -chunk.Components on the namespace and -chunk.SizeBytes on the repository for each committed purge chunk. This branch leaves components_count to the reap, so the two never move the same counter for the same row. !1916 (merged) is unmerged, so this note rests on its current diff. Re-check it against the merged form before this branch merges. !1916 (merged) also edits docs/testing/e2e/npm.md, at the e2e.npm.lifecycle.repository-storage-counters row on line 185, next to this branch's row on line 187. A rebase can need a manual resolution in that table.

npm remote fill emits nothing yet. An npm remote cache fill raises no counter today, and issue #834 (closed) tracks it. Until that lands, an npm remote repository's counters can read low between reconciliation passes. docs/specs/S22-storage-accounting.md:149 states that no counter column carries a non-negative CHECK and that a transient negative is tolerated, so this is not a spec breach.

Two spec sentences are already false, and this branch does not touch them. docs/specs/S22-storage-accounting.md:1007 and :1009 say that the npm and Maven remote soft_deleted_at columns have no writer a request can reach. They also say that the first writer marks the package row alone. Both are already false on origin/main, which serves the remote version mark at internal/managementapi/version_delete.go:237 and :356. This merge request changes no file under docs/specs/. !1973 (merged) removes those sentences.

Deviation from the plan guardrail

The project guardrail asks for a plan merge request before any implementation merge request opens. This is a feat issue and no plan file records it.

What remains

This merge request does not close the issue. The six single-target remote DELETE routes listed above still emit no counter, and part 2 covers them.

Database Review Evidence

Queries

Note

Plans are from EXPLAIN (ANALYZE, BUFFERS) against an ephemeral PostgreSQL 17.10 container (matching GL_PG_CURR_VERSION from .gitlab-ci-other-versions.yml), with synthesized seed data rolled back per query and the container torn down at the end of the run. Numbers reflect moderate cardinality and do not capture production-scale effects. See Database review evidence for seed sizing, methodology, and the anomalies the skill flags. Expand each row's details for the seed shape, rendered SQL, bound args, and raw plan.

Migration mode did not run. git diff --name-only --diff-filter=ACMR origin/main...HEAD -- internal/datastore/migrations/sql/ is empty at c14b901d8, and empty again at the tip bb5483ec9.

The plans in this section were measured at c14b901d8, which a later rebase replaced. That commit is no longer an ancestor of the tip, and the evidence is still live: none of the six statements changed after it. The two eviction files gained one comment each, and internal/datastore/repo_blob_references.go is untouched.

Method Plan node Index Rows (plan / actual) Cost Time Buffers (hit / read) Partitions
datastore.MavenRemoteEvictor.fileFreedSizeTx (repository resolve) Nested Loop unique_maven_remote_versions_id_pkg_id_ns_id, pk_maven_remote_packages 1 / 1 16.61 0.039ms 6 / 0 1/64, 1/64
datastore.NpmRemoteEvictor.fileFreedSizeTx (repository resolve) Nested Loop pk_npm_remote_versions, pk_npm_remote_packages 1 / 1 16.62 0.021ms 6 / 0 1/64, 1/64
datastore.countLiveMavenRemoteVersionsStmt Aggregate index_maven_remote_versions_on_ns_id_pkg_id_last_downloaded_at 1 / 1 93.17 0.069ms 5 / 0 1/64
datastore.countLiveNpmRemoteVersionsStmt Aggregate index_npm_remote_versions_on_ns_id_pkg_id_size_bytes 1 / 1 93.17 0.106ms 5 / 0 1/64
datastore.mavenRemoteRepoStillReferencesBlobStmt Result index_maven_remote_files_on_ns_id_blob_sha256, pk_maven_remote_packages 1 / 1 16.62 0.024ms 6 / 0 1/64, 1/64
datastore.npmRemoteRepoStillReferencesBlobStmt Result index_npm_remote_files_on_ns_id_blob_sha256, pk_npm_remote_versions, pk_npm_remote_packages, index_npm_remote_metadata_files_on_ns_id_blob_sha256 1 / 1 33.62 0.034ms 11 / 0 1/64, 1/64, 1/64, 1/64

All six statements prune to one partition of 64 on every table they touch, because each carries a namespace_id equality. No statement reaches a Seq Scan, a Sort, or a partition fan-out. The plan-to-actual ratio stays at or below 1.24x.

Seed shape (one namespace, shared by all six transactions): namespaces=1, repositories=4, blob_storage_blobs=16000, blob_storage_attachments=16000, blob_storage_blobs_by_namespace=16000, maven_remote_repositories=2, maven_remote_packages=2002, maven_remote_versions=6251, maven_remote_files=5500, npm_remote_repositories=2, npm_remote_packages=5002, npm_remote_versions=6251, npm_remote_files=5500, npm_remote_metadata_files=5000

The seed gives each format two remote repositories under one namespace. Repository A is the deleting repository. Repository B holds the digest the probe binds, so the probe finds a candidate row and the repository leg then rejects it. This is the shape the accounting contract describes, and it is the most work an EXISTS arm can do before it answers false.

datastore.MavenRemoteEvictor.fileFreedSizeTx (repository resolve)

Summary: The plan matches the method's intent, which is one row lookup that turns a version id into the repository the freed-size probe scopes on. The planner reads the version through an Index Only Scan on the three-column unique index, then the package through its primary key, and prunes both tables to one partition of 64. No anomalies.

Seed shape: maven_remote_versions=6251, maven_remote_packages=2002 (plus the shared ancestors listed above)

Rendered SQL:

SELECT mrp.maven_remote_repository_id FROM maven_remote_versions v
JOIN maven_remote_packages mrp ON mrp.namespace_id = v.namespace_id AND mrp.id = v.maven_remote_package_id
WHERE v.namespace_id = $1 AND v.id = $2

Bound args: [b59e7c22-e1d2-42b4-a50b-0afbe5910be8, 03f22d1f-17e5-44bb-9303-f22a57564709]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Nested Loop  (cost=0.56..16.61 rows=1 width=16) (actual time=0.037..0.039 rows=1 loops=1)
   Buffers: shared hit=6
   ->  Index Only Scan using maven_remote_versions_p52_id_maven_remote_package_id_namesp_idx on maven_remote_versions_p52 v  (cost=0.28..8.30 rows=1 width=32) (actual time=0.028..0.029 rows=1 loops=1)
         Index Cond: ((id = '03f22d1f-17e5-44bb-9303-f22a57564709'::uuid) AND (namespace_id = 'b59e7c22-e1d2-42b4-a50b-0afbe5910be8'::uuid))
         Heap Fetches: 1
         Buffers: shared hit=3
   ->  Index Scan using maven_remote_packages_p52_pkey on maven_remote_packages_p52 mrp  (cost=0.28..8.30 rows=1 width=48) (actual time=0.007..0.007 rows=1 loops=1)
         Index Cond: ((id = v.maven_remote_package_id) AND (namespace_id = 'b59e7c22-e1d2-42b4-a50b-0afbe5910be8'::uuid))
         Buffers: shared hit=3
 Planning:
   Buffers: shared hit=218
 Planning Time: 1.134 ms
 Execution Time: 0.063 ms

Timings: planning 1.134ms, execution 0.063ms, total 1.197ms.

datastore.NpmRemoteEvictor.fileFreedSizeTx (repository resolve)

Summary: The plan matches the method's intent, and it is the npm twin of the Maven resolve above. Both tables answer from their primary key and both prune to one partition of 64. No anomalies.

Seed shape: npm_remote_versions=6251, npm_remote_packages=5002 (plus the shared ancestors listed above)

Rendered SQL:

SELECT nrp.npm_remote_repository_id FROM npm_remote_versions v
JOIN npm_remote_packages nrp ON nrp.namespace_id = v.namespace_id AND nrp.id = v.npm_remote_package_id
WHERE v.namespace_id = $1 AND v.id = $2

Bound args: [c06cb994-971d-4f91-b1f8-d3982077e780, c3d3a99b-faf6-4b78-acd8-8d04ae70fe88]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Nested Loop  (cost=0.56..16.62 rows=1 width=16) (actual time=0.020..0.021 rows=1 loops=1)
   Buffers: shared hit=6
   ->  Index Scan using npm_remote_versions_p41_pkey on npm_remote_versions_p41 v  (cost=0.28..8.30 rows=1 width=32) (actual time=0.012..0.013 rows=1 loops=1)
         Index Cond: ((id = 'c3d3a99b-faf6-4b78-acd8-8d04ae70fe88'::uuid) AND (namespace_id = 'c06cb994-971d-4f91-b1f8-d3982077e780'::uuid))
         Buffers: shared hit=3
   ->  Index Scan using npm_remote_packages_p41_pkey on npm_remote_packages_p41 nrp  (cost=0.28..8.30 rows=1 width=48) (actual time=0.006..0.006 rows=1 loops=1)
         Index Cond: ((id = v.npm_remote_package_id) AND (namespace_id = 'c06cb994-971d-4f91-b1f8-d3982077e780'::uuid))
         Buffers: shared hit=3
 Planning:
   Buffers: shared hit=201 read=1
 Planning Time: 1.026 ms
 Execution Time: 0.041 ms

Timings: planning 1.026ms, execution 0.041ms, total 1.067ms.

datastore.countLiveMavenRemoteVersionsStmt

Summary: The plan matches the method's intent, which is the hidden-version count the package mark reports. The soft_deleted_at IS NULL leg is served by the index predicate rather than by a heap filter, because the planner picks a partial index whose own WHERE is that leg. The Bitmap Heap Scan sits above the index because the seeded pages carry no visibility-map bits inside the seeding transaction, so an Index Only Scan is unavailable here and is available on vacuumed pages; the scan node estimates 248 rows against 200 actual, a ratio of 1.24x.

Seed shape: maven_remote_versions=6251 in the pruned partition, of which the bound package holds 200 live rows and 50 marked rows

Rendered SQL:

SELECT COUNT(*)
FROM public.maven_remote_versions
WHERE ((maven_remote_versions.namespace_id = $1::uuid) AND (maven_remote_versions.maven_remote_package_id = $2::uuid)) AND (maven_remote_versions.soft_deleted_at IS NULL);

Bound args: [bd8b60a5-ccc1-4cd5-bf01-a74604052d1f, 6c0cdc5b-7073-4f61-a1ce-9fc10acaed68]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Aggregate  (cost=93.16..93.17 rows=1 width=8) (actual time=0.068..0.069 rows=1 loops=1)
   Buffers: shared hit=5
   ->  Bitmap Heap Scan on maven_remote_versions_p12 maven_remote_versions  (cost=10.82..92.54 rows=248 width=0) (actual time=0.040..0.056 rows=200 loops=1)
         Recheck Cond: ((namespace_id = 'bd8b60a5-ccc1-4cd5-bf01-a74604052d1f'::uuid) AND (maven_remote_package_id = '6c0cdc5b-7073-4f61-a1ce-9fc10acaed68'::uuid) AND (soft_deleted_at IS NULL))
         Heap Blocks: exact=3
         Buffers: shared hit=5
         ->  Bitmap Index Scan on maven_remote_versions_p12_namespace_id_maven_remote_packag_idx2  (cost=0.00..10.76 rows=248 width=0) (actual time=0.031..0.031 rows=200 loops=1)
               Index Cond: ((namespace_id = 'bd8b60a5-ccc1-4cd5-bf01-a74604052d1f'::uuid) AND (maven_remote_package_id = '6c0cdc5b-7073-4f61-a1ce-9fc10acaed68'::uuid))
               Buffers: shared hit=2
 Planning:
   Buffers: shared hit=114 read=1
 Planning Time: 0.763 ms
 Execution Time: 0.095 ms

The partition index maven_remote_versions_p12_namespace_id_maven_remote_packag_idx2 belongs to index_maven_remote_versions_on_ns_id_pkg_id_last_downloaded_at, which is (namespace_id, maven_remote_package_id, last_downloaded_at NULLS FIRST) WHERE (soft_deleted_at IS NULL). Three other partial indexes on the same two leading columns cost the same, so the choice among them carries no meaning.

Timings: planning 0.763ms, execution 0.095ms, total 0.858ms.

datastore.countLiveNpmRemoteVersionsStmt

Summary: The plan matches the method's intent and is the npm twin of the Maven count above, with the same partial-index predicate and the same 1.24x estimate ratio. No anomalies.

Seed shape: npm_remote_versions=6251 in the pruned partition, of which the bound package holds 200 live rows and 50 marked rows

Rendered SQL:

SELECT COUNT(*)
FROM public.npm_remote_versions
WHERE ((npm_remote_versions.namespace_id = $1::uuid) AND (npm_remote_versions.npm_remote_package_id = $2::uuid)) AND (npm_remote_versions.soft_deleted_at IS NULL);

Bound args: [7cce9ec2-8e73-4d0d-a0f3-22b3a3eba5d6, 08b76dac-31ff-4cec-8ca2-6c9e799dbca2]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Aggregate  (cost=93.16..93.17 rows=1 width=8) (actual time=0.105..0.106 rows=1 loops=1)
   Buffers: shared hit=5
   ->  Bitmap Heap Scan on npm_remote_versions_p38 npm_remote_versions  (cost=10.82..92.54 rows=248 width=0) (actual time=0.062..0.087 rows=200 loops=1)
         Recheck Cond: ((namespace_id = '7cce9ec2-8e73-4d0d-a0f3-22b3a3eba5d6'::uuid) AND (npm_remote_package_id = '08b76dac-31ff-4cec-8ca2-6c9e799dbca2'::uuid) AND (soft_deleted_at IS NULL))
         Heap Blocks: exact=3
         Buffers: shared hit=5
         ->  Bitmap Index Scan on npm_remote_versions_p38_namespace_id_npm_remote_package_id_idx1  (cost=0.00..10.76 rows=248 width=0) (actual time=0.050..0.050 rows=200 loops=1)
               Index Cond: ((namespace_id = '7cce9ec2-8e73-4d0d-a0f3-22b3a3eba5d6'::uuid) AND (npm_remote_package_id = '08b76dac-31ff-4cec-8ca2-6c9e799dbca2'::uuid))
               Buffers: shared hit=2
 Planning:
   Buffers: shared hit=100 read=1
 Planning Time: 1.166 ms
 Execution Time: 0.146 ms

The partition index npm_remote_versions_p38_namespace_id_npm_remote_package_id_idx1 belongs to index_npm_remote_versions_on_ns_id_pkg_id_size_bytes, which is (namespace_id, npm_remote_package_id, size_bytes DESC) WHERE (soft_deleted_at IS NULL).

Timings: planning 1.166ms, execution 0.146ms, total 1.312ms.

datastore.mavenRemoteRepoStillReferencesBlobStmt

Summary: The plan matches the probe's intent, which is a repository-scoped last-reference test over the row set recomputeMavenRemoteFilesSizeStmt walks. The digest leg answers from index_maven_remote_files_on_ns_id_blob_sha256, and the repository leg then rejects the sibling repository's row through the package primary key, so the probe answers false in 6 buffer hits. Both tables prune to one partition of 64. No anomalies.

Seed shape: maven_remote_files=5500 in the pruned partition (5000 under repository A, 500 under repository B), maven_remote_packages=2002

Rendered SQL:

SELECT EXISTS (
	SELECT 1 FROM maven_remote_files mrf
	JOIN maven_remote_packages mrp
		ON mrp.namespace_id = mrf.namespace_id AND mrp.id = mrf.maven_remote_package_id
	WHERE mrf.namespace_id = $1 AND mrp.maven_remote_repository_id = $2 AND mrf.blob_sha256 = $3
)

Bound args: [4d24629d-f6db-4a87-8b8b-35e804232284, 642e8916-f995-4a14-a281-2b7f4a8d1263, \x0000000000000000000000000000000000000000000000000000000000001389]

The digest belongs to a maven_remote_files row of the sibling remote repository, not of repository A. This is the bind that makes the probe do the most work, because the index returns a candidate row and only the repository leg rejects it.

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Result  (cost=16.61..16.62 rows=1 width=1) (actual time=0.023..0.024 rows=1 loops=1)
   Buffers: shared hit=6
   InitPlan 1
     ->  Nested Loop  (cost=0.56..16.61 rows=1 width=0) (actual time=0.021..0.022 rows=0 loops=1)
           Buffers: shared hit=6
           ->  Index Scan using maven_remote_files_p07_namespace_id_blob_sha256_idx on maven_remote_files_p07 mrf  (cost=0.28..8.30 rows=1 width=32) (actual time=0.012..0.013 rows=1 loops=1)
                 Index Cond: ((namespace_id = '4d24629d-f6db-4a87-8b8b-35e804232284'::uuid) AND (blob_sha256 = '\x0000000000000000000000000000000000000000000000000000000000001389'::bytea))
                 Buffers: shared hit=3
           ->  Index Scan using maven_remote_packages_p07_pkey on maven_remote_packages_p07 mrp  (cost=0.28..8.30 rows=1 width=32) (actual time=0.007..0.007 rows=0 loops=1)
                 Index Cond: ((id = mrf.maven_remote_package_id) AND (namespace_id = '4d24629d-f6db-4a87-8b8b-35e804232284'::uuid))
                 Filter: (maven_remote_repository_id = '642e8916-f995-4a14-a281-2b7f4a8d1263'::uuid)
                 Rows Removed by Filter: 1
                 Buffers: shared hit=3
 Planning:
   Buffers: shared hit=319 read=2
 Planning Time: 2.025 ms
 Execution Time: 0.044 ms

Timings: planning 2.025ms, execution 0.044ms, total 2.069ms.

datastore.npmRemoteRepoStillReferencesBlobStmt

Summary: The plan matches the probe's intent, which is a two-arm last-reference test over the row set recomputeNpmRemoteFilesSizeStmt unions. Each arm starts from its own (namespace_id, blob_sha256) index, walks up through primary keys, and prunes every one of the four tables to a single partition of 64. The tarball arm returns a candidate row that the repository leg rejects, and the packument arm returns none, so the packument arm's package lookup is marked never executed; the whole probe costs 11 buffer hits. No anomalies.

Seed shape: npm_remote_files=5500 in the pruned partition (5000 under repository A, 500 under repository B), npm_remote_metadata_files=5000, npm_remote_versions=6251, npm_remote_packages=5002

Rendered SQL:

SELECT EXISTS (
	SELECT 1 FROM npm_remote_files nrf
	JOIN npm_remote_versions nrv
		ON nrv.namespace_id = nrf.namespace_id AND nrv.id = nrf.npm_remote_version_id
	JOIN npm_remote_packages nrp
		ON nrp.namespace_id = nrv.namespace_id AND nrp.id = nrv.npm_remote_package_id
	WHERE nrf.namespace_id = $1 AND nrp.npm_remote_repository_id = $2 AND nrf.blob_sha256 = $3
) OR EXISTS (
	SELECT 1 FROM npm_remote_metadata_files nrmf
	JOIN npm_remote_packages nrp
		ON nrp.namespace_id = nrmf.namespace_id AND nrp.id = nrmf.npm_remote_package_id
	WHERE nrmf.namespace_id = $1 AND nrp.npm_remote_repository_id = $2 AND nrmf.blob_sha256 = $3
)

Bound args: [8b317595-c456-42cb-a5c4-781004852eca, 1972cec1-0c27-4dae-8027-88d28775291c, \x0000000000000000000000000000000000000000000000000000000000007531]

The digest belongs to an npm_remote_files row of the sibling remote repository. A tarball digest and a packument digest name different objects, so a bind that hits both arms is not a shape production produces.

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Result  (cost=33.61..33.62 rows=1 width=1) (actual time=0.032..0.034 rows=1 loops=1)
   Buffers: shared hit=11
   InitPlan 1
     ->  Nested Loop  (cost=0.85..17.00 rows=1 width=0) (actual time=0.024..0.025 rows=0 loops=1)
           Buffers: shared hit=9
           ->  Nested Loop  (cost=0.56..16.62 rows=1 width=32) (actual time=0.017..0.019 rows=1 loops=1)
                 Buffers: shared hit=6
                 ->  Index Scan using npm_remote_files_p26_namespace_id_blob_sha256_idx on npm_remote_files_p26 nrf  (cost=0.28..8.30 rows=1 width=32) (actual time=0.010..0.011 rows=1 loops=1)
                       Index Cond: ((namespace_id = '8b317595-c456-42cb-a5c4-781004852eca'::uuid) AND (blob_sha256 = '\x0000000000000000000000000000000000000000000000000000000000007531'::bytea))
                       Buffers: shared hit=3
                 ->  Index Scan using npm_remote_versions_p26_pkey on npm_remote_versions_p26 nrv  (cost=0.28..8.30 rows=1 width=48) (actual time=0.006..0.006 rows=1 loops=1)
                       Index Cond: ((id = nrf.npm_remote_version_id) AND (namespace_id = '8b317595-c456-42cb-a5c4-781004852eca'::uuid))
                       Buffers: shared hit=3
           ->  Index Scan using npm_remote_packages_p26_pkey on npm_remote_packages_p26 nrp  (cost=0.28..0.37 rows=1 width=32) (actual time=0.005..0.005 rows=0 loops=1)
                 Index Cond: ((id = nrv.npm_remote_package_id) AND (namespace_id = '8b317595-c456-42cb-a5c4-781004852eca'::uuid))
                 Filter: (npm_remote_repository_id = '1972cec1-0c27-4dae-8027-88d28775291c'::uuid)
                 Rows Removed by Filter: 1
                 Buffers: shared hit=3
   InitPlan 2
     ->  Nested Loop  (cost=0.56..16.62 rows=1 width=0) (actual time=0.006..0.006 rows=0 loops=1)
           Buffers: shared hit=2
           ->  Index Scan using npm_remote_metadata_files_p26_namespace_id_blob_sha256_idx on npm_remote_metadata_files_p26 nrmf  (cost=0.28..8.30 rows=1 width=32) (actual time=0.006..0.006 rows=0 loops=1)
                 Index Cond: ((namespace_id = '8b317595-c456-42cb-a5c4-781004852eca'::uuid) AND (blob_sha256 = '\x0000000000000000000000000000000000000000000000000000000000007531'::bytea))
                 Buffers: shared hit=2
           ->  Index Scan using npm_remote_packages_p26_pkey on npm_remote_packages_p26 nrp_1  (cost=0.28..8.30 rows=1 width=32) (never executed)
                 Index Cond: ((id = nrmf.npm_remote_package_id) AND (namespace_id = '8b317595-c456-42cb-a5c4-781004852eca'::uuid))
                 Filter: (npm_remote_repository_id = '1972cec1-0c27-4dae-8027-88d28775291c'::uuid)
 Planning:
   Buffers: shared hit=522 read=1
 Planning Time: 3.111 ms
 Execution Time: 0.085 ms

Timings: planning 3.111ms, execution 0.085ms, total 3.196ms.

Row-set agreement with the recompute: each probe walks the tables its own doc comment names, and neither filters soft_deleted_at, matching recomputeMavenRemoteFilesSizeStmt and recomputeNpmRemoteFilesSizeStmt in internal/datastore/reconcile_repository.go. The probes stop one join short of the recompute: a recompute keys on maven_remote_repositories.repository_id, and a probe keys on maven_remote_packages.maven_remote_repository_id, which the evictor resolved in the same transaction. (namespace_id, repository_id) is unique on both remote repository tables, so the two predicates select the same repository and the two row sets agree.

Pipeline history

Two pipelines on this branch reported a failure on 2026-08-27. Both came from infrastructure faults outside this repository, and neither came from the diff. A retry corrected both faults, and no code change was necessary.

Bridge fault, 2026-08-27. Pipeline 2794755350 on 932aa721a reported failed, but all 46 jobs succeeded and a scope[]=failed query returned zero jobs. The failure was a bridge, and the jobs list does not show bridges. The bridge is build-jobs in the validate stage, and it has allow_failure: false. It ended after 0.45 seconds, with failure_reason: data_integrity_failure and downstream: null. GitLab did not create the child pipeline, so no code from this branch ran. A retry of the bridge created the downstream pipeline 2794803832 without error, which identifies the fault as transient.

Binary cache fault, 2026-08-27. The downstream pipeline 2794803832 then failed in build-current-artifact-registry-amd64, with script_failure and allow_failure: false. The binary cache at cache.nixos.org truncated a transfer, then answered the resumed range request with HTTP 416. As a result, Nix did not substitute skopeo-1.21.0, and its derivation failed. The Nix output states that this problem "usually happens due to networking issues". No Go code in this merge request reaches the skopeo derivation. A retry of that job was successful, which identifies the fault as transient.

Outcome. Pipeline 2794755350 ended at 2026-08-27T01:43:47Z with the status success and zero failed jobs. In that pipeline, the bridge build-jobs has the status success, and it created the downstream pipeline 2794803832. The downstream pipeline has the status success and zero failed jobs.

Related to #775 (closed)

This is a bot message 🤖 — /smurfit

Edited by Pawel Rozlach

Merge request reports