fix(container): emit accounting deltas on remote cache eviction

What this delivers

The container remote cache eviction now emits the counter delta it moves. repositories.artifacts_count falls at the eviction instead of waiting for the next reconciliation pass.

The scope is the Container arm of issue #775 (closed), settled before implementation. It covers the two bulk remote eviction arms, image and manifest, plus the two Container single-target management DELETE routes. !2003 (merged) already delivered the npm and Maven bulk arms. The six npm and Maven single-target remote DELETE routes stay out of this merge request.

Only artifacts_count moves. repositories.size_bytes and namespace_statistics.components_count carry no soft-delete predicate on this path, so an eviction leaves both at their pre-eviction values.

Cause

The defect sits in the datastore composers, not at the handlers. ContainerRemoteEvictor.EvictContainerRemoteImage ran its marker UPDATE on a caller-supplied qrm.DB and never read RowsAffected. evictContainerRemoteManifestTx discarded RowsAffected too. No caller therefore had a way to separate a committed tombstone from a lost mark race, and none held the hidden-manifest count. The four consumer sites had nothing to emit, and artifacts_count stayed high until reconciliation. A fix at the handlers alone is not possible: the value they must emit does not exist below them.

The executor parameter is the second half of the cause. Both production bindings passed the pool, so a count added on that seam lands as a second autocommit statement, separate from the mark. A failure between the two statements leaves the mark committed and its decrement never emitted, because the next call reports marked=false. The mark and the count therefore have to commit or roll back together, which the composer can only guarantee by owning the transaction itself.

The change

EvictContainerRemoteImage drops its db qrm.DB parameter, owns its transaction, and returns (marked, hiddenManifests, error). EvictContainerRemoteManifest returns (parents, marked, error). A new statement counts the live cached manifests inside the marking transaction, so the count and the mark commit together. Each of the four consumer sites emits deleteCounterTargetTombstone when marked is true, and emits nothing when it is false. A refused manifest eviction and a repeat call therefore move no counter.

cmd/artifact-registry/container_remote_bulk_evictor.go loses containerRemoteBulkEvictorBinding and containerRemoteBulkEvictionStore. Without the db parameter the composer satisfies the bulk interface directly, the way npm binds its remote evictor.

Coverage

Source: the nine acceptance items derived in the issue enrichment note for work item 775, each row naming the artifact the item was read from.

#	Acceptance item	Tests
1	A bulk pass marking a cached image falls artifacts_count by the live container_remote_manifests rows the mark hid, counted inside the mark's own transaction (enrichment acceptance item 1, derived from the issue body's Container counter table and recomputeContainerRemoteManifestsStmt)	TestContainerRemoteEvictor_EvictContainerRemoteImage_ReportsWhatTheMarkHid (7 subtests, including the transaction's report survives the reap a recount would not), TestContainerBulkWorker_RemoteArm_EmitsTheDeltasItsTransactionsMoved/{subset,delete_all}_images_carry_the_hidden-manifest_counts
2	A bulk pass marking a cached manifest whose blocking check came back empty falls artifacts_count by one (enrichment item 2, same source)	TestContainerRemoteEvictor_EvictContainerRemoteManifest_ReportsWhetherTheMarkApplied/a_mark_that_applied_reports_it, TestContainerBulkWorker_RemoteArm_EmitsTheDeltasItsTransactionsMoved/{subset,delete_all}_manifests_carry_one_each, /a_chain_cleared_in_one_pass_emits_once_per_manifest_that_left_the_count
3	A refused manifest eviction moves no counter and still answers its 409 (enrichment item 3, derived from the issue body's counter table)	TestContainerRemoteEvictor_EvictContainerRemoteManifest_ReportsWhetherTheMarkApplied/a_refused_eviction_reports_its_parents_and_no_mark, TestContainerBulkWorker_RemoteArm_EmitsTheDeltasItsTransactionsMoved/a_manifest_refused_for_its_live_parents_emits_nothing, TestContainerRemoteManifestDelete_NonCommittedOutcomesEmitNothing/a_refusal_answers_409_and_emits_nothing
4	A repeated eviction of an already-marked cached image or manifest moves no counter (enrichment item 4, same source)	..._ReportsWhatTheMarkHid/{a_repeat_mark_reports_no_write_and_no_count,an_id_no_row_matches_reports_no_write_and_no_count}, ..._ReportsWhetherTheMarkApplied/{a_repeat_mark_reports_no_write,an_id_no_row_matches_reports_no_write}, TestContainerBulkWorker_RemoteArm_EmitsTheDeltasItsTransactionsMoved/{a_re-marked_cached_image_emits_nothing,a_re-marked_cached_manifest_emits_nothing}
5	repositories.size_bytes does not move on any container remote eviction, and a reconciliation immediately afterwards does not move it either (enrichment item 5, derived from recomputeContainerRemoteBlobsSizeStmt's absent predicate and its own doc)	TestContainerRemoteEviction_MovesTheReconciledArtifactCountAndNothingElse (both subtests), plus the deltaSize assertions in TestContainerBulkWorker_RemoteArm_EmitsTheDeltasItsTransactionsMoved, TestContainerRemoteImageDelete_EmitsTheTombstoneDeltas and TestContainerRemoteManifestDelete_EmitsTheTombstoneDelta
6	namespace_statistics.components_count does not move on any container remote eviction (enrichment item 6, derived from recomputeNamespaceComponentsCountStmt's absent predicate)	TestContainerRemoteEviction_MovesTheReconciledArtifactCountAndNothingElse (both subtests), plus the namespaceEmits() assertions in the three emit suites above
7	A single-target DELETE of a cached container image moves artifacts_count by the hidden-manifest count, and one that lost the mark race moves nothing (the single-target image pair, which this merge request scope includes)	TestContainerRemoteImageDelete_EmitsTheTombstoneDeltas, TestContainerRemoteImageDelete_NonCommittedOutcomesEmitNothing (3 subtests)
8	A single-target DELETE of a cached container manifest by digest moves artifacts_count by one when the mark applied, and by nothing when it was refused or matched no row (enrichment item 8, same pair)	TestContainerRemoteManifestDelete_EmitsTheTombstoneDelta, TestContainerRemoteManifestDelete_NonCommittedOutcomesEmitNothing (4 subtests)
9	After an eviction the persisted artifacts_count equals what a reconciliation pass recomputes for the same repository: no climb-back and no second decrement (enrichment item 9, derived from the reconciliation rule the issue anchors on — "the reconciled repositories.artifacts_count equals the live count (tombstoned artifacts excluded), while repositories.size_bytes and namespace_statistics.components_count include them")	TestContainerRemoteEviction_MovesTheReconciledArtifactCountAndNothingElse (both subtests, comparing each mark's report against RecomputeArtifactsCount, RecomputeSizeBytes and RecomputeComponentsCount taken before and after)

Two suites landed after the test commit, and they strengthen rows 1, 2, 7, 8 and 9:

• TestIntegration_ContainerBulkRemote_EmittedDeltasAgreeWithTheRecompute, the container twin of the Maven case. It seeds two images so that both eviction arms contribute. An image mark that recounted its whole subtree disagrees with the recompute.
• TestContainerRemoteEvictionIntegration_SingleTargetDeletesEmitTheDeltasTheirMarksMoved, three subtests against RecomputeArtifactsCount before and after the mark.

Runtime evidence

The service ran locally from this branch against a seeded remote container repository. The run drove all ten exercises on the list. artifacts_count moved on every committed eviction and on nothing else. size_bytes held at its start value across all six remote evictions, and namespace_statistics.components_count held at 1. A reconciliation pass after the evictions read the same artifacts_count the emits left, so there was no climb-back and no second decrement.

One prediction on that list was wrong, and this branch does not change the behavior behind it. A repeat image DELETE answers 404, because findContainerRemoteImageForRequest gates ahead of the evictor. The marked == false arm is therefore reachable only by a genuine race, which is why no test drives it directly.

End-to-end scenario catalogs

docs/testing/e2e/docker.md and docs/testing/e2e/oci.md both gain a remote leg on their delete-storage-counters scenario. The lifecycle row now states what the image, manifest and tag routes move against a remote repository, and where size_bytes waits. The usage-data row gains the carve-out !2003 (merged) wrote for maven.md: the journey's remote leg emits no artifact_registry_artifact_deleted. docs/testing/e2e/maven.md and docs/testing/e2e/npm.md stay untouched, because their published exclusion sentence is still true under this scope.

Diff size

Measured at HEAD 0b0736fca against the merge base 21cc02127 on 2026-09-01T19:49Z, with git diff --shortstat 21cc02127 0b0736fca and git diff --numstat 21cc02127 0b0736fca.

File group	Files	Added	Removed	Net
Production Go	10	240	221	+19
Tests	17	1445	320	+1125
Documentation	3	25	13	+12
Total	30	1710	554	+1156

Three of the 17 test files are new and 14 are edits. No production file is new. The production group is close to flat, at +19 lines over 10 files. The tests and the documentation carry the rest of the diff. These figures are true at the two shas above, and a later push or rebase of the branch moves them.

Why the production group is close to flat

The +19 is a net figure, not a small change. The group adds a counting statement, the four consumer-site emits, and four INFO lines on the path where a mark did not apply. A comment deletion of about the same size offsets all of it.

The comment-caps gate charges the comment segments a diff touches. Both seam docs were far over cap on origin/main already. EvictContainerRemoteImage's doc block was 37 lines, of which 33 count against the cap. EvictContainerRemoteManifest's was 55 lines, of which 49 count. Bare // separator lines do not count, which is why each counted total is below its block's line count. The cap for an exported top-level doc is 3 lines. The signature change altered what both docs promise, so the branch had to touch them, and the gate then charged both blocks whole. The large comment deletion in this diff is that compression. It is inherited debt rather than information this branch removed.

One claim had no Go site left with budget: the manifest arm's blocking check takes no row locks and its transaction runs at READ COMMITTED. That paragraph moved into docs/dev/storage-accounting.md, which carries no comment cap.

Why the documentation change is this large

!1973 (merged) merged during this work and rewrote about 1,300 lines of docs/dev/storage-accounting.md. Its new text asserts in five places that the container remote eviction emits nothing, and this branch corrects all five. One neighboring clause stays in place: the six npm and Maven single-target routes issue no delta, and that half of the sentence remains true.

Why a split does not help

The two datastore composers change signature, and the four consumer sites read the values that change makes available. A split leaves either composers that report values nobody reads, or handlers that read values that do not exist. The tests and the documentation travel with the behavior they pin.

Merge order

These merge results date from 2026-09-01, at HEAD 8ef7a5126 and origin/main f51f4db3e. The branch then moved to 0b0736fca, on merge base 21cc02127, and this refresh did not take them again. Every head came from refs/merge-requests/<iid>/head, read by direct per-iid GET, because the state=opened list serves stale rows. Each pair was tested with git merge-tree --write-tree. git merge-tree --write-tree origin/main HEAD is clean.

Three files are the contact points.

internal/datastore/query_names.go

Seven other open merge requests touch it: !2040 (merged), !2123 (merged), !2135 (merged), !2136 (merged), !2145 (merged), !2146 (merged) and !2162 (merged). All seven merge cleanly with this branch. None of them adds a name inside the container_remote_manifests block this branch adds to, so the contact is additive.

The file is unchanged between the merge base 21cc02127 and origin/main. An earlier version of this section said it gained 9 lines on main, across five commits. That was true at the older merge base 98a7b2612, and the rebase onto 21cc02127 made those five commits ancestors of the base.

docs/dev/storage-accounting.md

Four other open merge requests touch it: !2040 (merged), !2123 (merged), !2162 (merged) and !2169 (merged). !2123 (merged) is the one conflict, and it is one table row. Both branches edit the Maven remote cache fill row. This branch removes the clause that says the container eviction does not emit. The !2123 (merged) edit removes npm from the clause that says the cache fills emit nothing. Both edits are wanted, so the resolution keeps both. !2040 (merged), !2162 (merged) and !2169 (merged) merge cleanly.

!2123 (merged) merges cleanly into the merge base and into origin/main, so the conflict is the two branches' own content rather than drift on main.

This branch also removes the #775 row from the known-gaps table, and carries the rows around it as unchanged context. That resolves cleanly while those neighboring rows stay where they are. The clean merge-tree above says they do, at the two shas named. It is a measurement rather than a property of the diff, so it holds until one of the four claimants moves a neighboring row.

docs/testing/e2e/docker.md and docs/testing/e2e/oci.md

!2052 carries the branch sahmed/e2e-caproni-authz into main, open at head 2353ea30e. !2130 (merged) merged into that branch on 2026-09-01, so the catalog content it added now sits on !2052. !2161 (merged) is open against the same branch, at head a5d103733c. Both conflict with this branch, on both catalog files.

The conflict is this branch's content. Both also conflict with origin/main on docs/testing/e2e/npm.md, which this branch does not touch. They carry that same conflict against 98a7b2612, the merge base at that measurement. Against this branch they add docker.md and oci.md, and the branch is the only difference between the two measurements. Whichever side lands second carries the resolution on both files. !2052 must resolve its npm.md conflict against main in any case.

!2103 (merged) and !2186 (merged) touch both catalog files too, and both merge cleanly with this branch.

!2106 (merged)

The signature conflict is gone. At head a2dc638103 !2106 (merged) no longer touches internal/datastore/container_remote_eviction_integration_test.go, and it merges cleanly with this branch. It still touches internal/managementapi/bulk_container_worker.go and that file's integration test, which this branch touches too, and both merge cleanly.

What a stalled Redis costs the bulk arms

The two bulk arms call the emit inline, on the worker goroutine. internal/accounting/emit.go bounds each Redis call at detachedEmitTimeout, which is 5 seconds. Lines 57-59 of that file state the design. The emit spawns nothing and sheds nothing, so where a caller waits is settled at the call site.

When Redis stalls, an applied entry costs about 5 seconds. containerBulkWorkTimeout gives one attempt 45 minutes, so one attempt applies about 540 entries under that condition. A delete_all pass over 20,000 entries then needs many attempts. The pass loses no work. Each entry commits on its own, and an applied entry leaves the scope predicate. The next attempt therefore resumes where the last one stopped. containerBulkWorkTimeout's own comment states that resume rule.

This is stated so a reviewer knows the behavior, not proposed as a change. Ten call sites in the npm, Maven and container bulk workers already call this emit inline on origin/main. emitDeleteCountersDetached is what the single-delete routes use, the two this branch wires included, and docs/dev/storage-accounting.md records that split.

Accepted deviations

Both items in this section are deliberate. Each one states what it leaves uncovered.

The branch is not squashed

This branch ships as a set of conventional commits, not as one squashed commit. Phase 5 attempted the squash and golangci-lint --new-from-rev HEAD refused it. The count is 20 commits, from git rev-list --count origin/main..origin/prozlach/issue-775-remote-eviction-container-deltas at 2026-09-01T19:49Z, against merge base 21cc02127. That figure is true at the head the command read, and a later push to the branch moves it.

No third validation pass

/validate-issue ran twice, both times at phase 4. The /smurfit skill defines phase 9 as "no second validate pass", and it records that skip as deliberate. Phase 8 then landed its fixes after the last validation pass, and no further pass covered them. Phase 8's /review-branch and its triage gate reviewed the final state of the branch. A validation pass is a different check. It compares the branch's own prose with its acceptance evidence. The phase-8 edits did not get that check.

Issue link

An earlier version of this section carried Related to, because the nine acceptance items came from the tooling rather than from the issue author. That reason left the decision to the author, and the author has now made it.

Closes #775 (closed)

Still open after this merge: the six npm and Maven single-target remote DELETE routes emit nothing, which is !2003 (merged)'s part 2, tracked at #1018 (closed)

This is a bot message 🤖 — /smurfit