chore(datastore): container remote reap (S20-A plan: 12/21) (!1704) · Merge requests · GitLab.org / ops / Artifact Registry

What this step delivers

Reap logic for container_remote_images and container_remote_manifests. Each reaper takes one TombstoneRow and removes that row's subtree. Every statement runs on the db handle the caller passes, so one chunk stays one transaction.

BlobStorageAttachmentStore.DeleteIfUnreferencedByContainerRemote is the new store arm. It removes a reaped row's blob_storage_attachments row in the same transaction, and only when no other container_remote_manifests row in the namespace still holds that attachment. container_remote_manifests.blob_storage_attachment_id is NOT NULL, and no referential action on that foreign key removes the attachment as its referencing row goes.

There are two reapers rather than one. Reaper carries a single Reap method, and the discovery scan that reaches a reap fixes the level it runs at.

No production code constructs either reaper yet. The chunk driver that will call a Reaper lands with internal/lifecycle at step 15.

The image reap is state-blind

The image reaper does not test the image row's own soft_deleted_at, and it tests no manifest's either. Its authority is the tombstone on the repository above it.

A gate on the image row's own soft_deleted_at would make this reap a no-op under its only route. A repository tombstone stamps the repositories row alone, so nothing marks the images beneath it and the walk down hands the reaper live rows. One expression also gates the child manifests, so nothing under the image would go either. A surviving image row then aborts the repository delete with SQLSTATE 23503.

Reap states that as a precondition the caller owes: the repository holding the image must already carry a tombstone when the row is dispatched. The precondition is stated rather than enforced, because NewContainerRemoteImageReaper and Reap are both exported and the type is constructible without the walk. A stated precondition is a claim a reviewer can check. A predicate that refuses every row the one route delivers is not, and it reads as though the reaper enforced something.

The precondition carries its own condition. It holds for as long as ScanTombstonedRepositories stays the only source of a dispatched TombstoneRow. If a second source arrives that can emit a row whose repository carries no tombstone, that source owes the precondition its own answer.

The child delete's EXISTS gate and the image-row delete still render one shared expression, so they cannot come apart over which image a chunk acts on.

One spec claim contradicted this, in two places, and this MR amends both. The Data Model inventory in docs/specs/S20-a-lifecycle-closed-beta.md said of the container remote-cache tables that the purger "reaps by the column, as it does for every other format", which the spec's own The repository-level walk is state-blind section already pointed away from. The amended bullet splits the claim by table: container_remote_manifests is reaped by the column, while a marked container_remote_images row is reaped exactly as an unmarked one, on the authority of the repository's own tombstone.

The second occurrence is in Hosted container needs one column, on container_images, where it read "the purger reaps those by the column like any other format". It was found by grepping the sentence rather than by trusting a line number, because the first amendment moved the line and this MR's later commits move it again. It now names each of the three tables: container_remote_manifests reaped by the column and the one of the three carrying a tombstone-discovery index, container_remote_images reaped state-blind on the authority of the repository's tombstone with the link to the section that states it, and container_remote_blobs reaped on container_remote_manifests's terms with each row's attachment removed in the same transaction. Striking the clause was rejected: naming two of the three would leave the same half-a-list gap one level down. The container_remote_blobs clause states what AC-19 asks for rather than that nothing reaps that table today: the deferral is this step's, it is recorded under One spec criterion is deferred and in the coverage table, and AC-19 stands as a requirement the spec still carries.

The sentence otherwise stands for the next reader of each section, which is why both are amended here rather than recorded only in this description.

`ReapTotals.SizeBytes` is zero in both reapers

The zero is a decision, not an omission.

Summing container_remote_manifests.size over the chunk is the reading the column's name invites, and it sits on the row being deleted. It was considered and rejected, on two grounds:

ADR-007 defines that column as a subtree footprint, not the row's own bytes, so a parent's size already covers its children's.
ADR-007 makes the repository counter a per-sha256 last-attachment test. The delta is therefore the summed size of the distinct blobs the subtree stopped reaching, over SELECT DISTINCT blob_sha256 rather than a per-row sum. A per-row sum double-counts whenever two reaped rows share a digest.

That duplicate is reachable rather than hypothetical. The only unique index on (namespace_id, container_remote_image_id, digest) is partial on soft_deleted_at IS NULL. So one image can hold a tombstoned row and a live row at one digest, and the state-blind image reap takes both in one chunk.

Nothing reads the field today. TombstoneRow carries a namespace id and a row id and no repository identifier, internal/lifecycle does not exist yet, and EmitRepoCounters has no production caller. That is what makes this a question about the comment rather than about correctness.

The file-head comment splits the two zeros, because the level each reaper is reached at differs. ContainerRemoteManifestReaper is reached as an artifact purge, which accounts for both counter deltas, so its zero is a stated gap. ContainerRemoteImageReaper is reached only from the walk down a tombstoned repository, and a repository purge owes no size delta, so its zero is correct.

Four rows corrected in the spec-coverage table

test-author produced the table below before the image reap became state-blind, in 9d35ef0a. Four of its rows named a test the branch no longer holds, so those four are corrected here rather than pasted unchanged. A reviewer diffing the table against 60a008f3's commit body finds exactly these four differences, and this section is the record of them.

Two things moved on the branch:

TestContainerRemoteImageReaper_KeysOnTheTombstone is now TestContainerRemoteImageReaper_KeysOnTheRowItIsHanded.
Its subtest a live image handed to the reaper is refused is replaced by a live image handed to the reaper is reaped, the positive twin of the gate the fix removed.

What each corrected row now says:

Row	Correction
AC-2	Split by table. `container_remote_manifests` stays covered. `container_remote_images` is marked not covered by design, and the state-blind behavior is cited positively.
AC-19	Test rename only. Both subtests the row cites still exist under the new name.
E-18	Test rename only. Both subtests the row cites still exist under the new name.
S-1	Rewritten to state each reaper's authority at its own level. "A reap with no tombstone behind it removes nothing" was true of both reapers, and it is now true of the manifest reaper only.

Every row about ContainerRemoteManifestReaper is untouched. That reaper kept its state conjunct, its test name and its refused subtest, so its rows are still accurate. The asymmetry between the two reapers is the point of the state-blind change. A blanket edit across both reapers adds a new false claim.

Every other row resolves against the branch.

Spec coverage

Spec: docs/specs/S20-a-lifecycle-closed-beta.md

Acceptance criteria

#	Criterion	Tests
AC-1	Eligible `repositories` tombstone discovered, ineligible one not; window non-zero and zero	Not this step. Step 5 owns the repository scan and its window cases.
AC-2	A `soft_deleted_at IS NULL` row is never discovered by any scan or walk, for every table in the reap inventory	Covered for `container_remote_manifests`: `TestContainerRemoteManifestReaper_KeysOnTheTombstone/a live manifest handed to the reaper is refused`. Not covered for `container_remote_images`, by design. That reap is state-blind on the row it is handed, pinned positively by `TestContainerRemoteImageReaper_KeysOnTheRowItIsHanded/a live image handed to the reaper is reaped`. What keeps a live image out of the reaper is the walk's own gate, which Step 5's `ScanTombstonedRepositories` owns. The other tables belong to Steps 5 to 13.
AC-3	The `repositories`, `npm_packages` and `container_images` scans use their index and prune to one partition	Not this step. Steps 5 and 7 own those `EXPLAIN` assertions.
AC-4	The version-level scans use their indexes and prune to one partition, per table	Not this step. Step 6 owns the five version-level scans, `container_remote_manifests` included.
AC-5	Re-running a purge on a fully reaped subtree is a no-op reporting success; a mid-walk abort leaves fewer rows	Reaper half covered: `TestContainerRemoteManifestReaper_Reap/re-reaping an already reaped manifest is a no-op that reports success`, `TestContainerRemoteImageReaper_Reap/re-reaping an already reaped image is a no-op that reports success`. The `gofail` mid-walk abort is Step 14's.
AC-6	The npm repository walk reaps live and already-tombstoned rows in one pass	Not this step (npm). The container-remote analogue is at AC-19.
AC-7	Every purge transaction is bounded; rows per transaction do not grow with the repository's contents	Reaper half covered: `TestContainerRemoteImageReaper_BoundsEachChunk`. The "more than one transaction" observation needs the chunk driver and is Step 15's.
AC-8	A purge job in a frozen namespace refuses retryably before any row is reaped	Not this step. Step 15a owns the write-serviceability gate.
AC-9	npm version reap with its files and attachments, through npm's own reference check	Not this step. Step 8.
AC-10	npm unpublish-then-republish leaves no orphan at the shared coordinate	Not this step. Step 8. The container-remote analogue is at AC-19.
AC-11	npm package reap with tags, metadata files and their attachments, no `23503`	Not this step. Step 8.
AC-12	`DeleteNpmMetadataFiles` removes each attachment in the same transaction	Not this step. Step 8.
AC-13	Maven version and package reap with their files and attachments	Not this step. Step 9.
AC-14	A container repository is reaped through `ContainerManifestDeleter` and `ContainerBlobUnlinker`, deleters observed	Not this step. Steps 13 and 14 own deleter observation.
AC-15	An index and the manifests it references are reaped parent-first, no `MANIFEST_REFERENCED` refusal	Not this step. Step 13.
AC-16	A tombstoned `container_images` row under a live repository is reaped through the same walk	Not this step. Step 13.
AC-17	The container repository walk reaps live and already-tombstoned images in one pass	Not this step. Step 14. The container-remote analogue at image level is at AC-19.
AC-18	A tombstoned remote-cache subtree is reaped for every remote table carrying the column, `container_remote_images` among them, a positive hit per table	`TestContainerRemoteImageReaper_Reap/reaps the image with the manifests beneath it, in both states`, `TestContainerRemoteImageReaper_Reap/an image with no manifests beneath it is reaped on its own`. The Maven and npm remote tables are Steps 10 and 11.
AC-19	`container_remote_manifests` and `container_remote_blobs` are reaped the same way with each attachment removed in the same transaction, and `container_remote_tags` and `container_remote_manifest_relationships` are cleared by walking the parent. A positive hit per table	`container_remote_manifests` covered: `TestContainerRemoteManifestReaper_Reap/removes the tombstoned manifest and its attachment`, `.../a rolled-back chunk leaves the manifest and its attachment in place`, `TestContainerRemoteImageReaper_Reap/a rolled-back chunk leaves the image and its subtree in place`, `TestContainerRemoteManifestReaper_SparesStillReferencedAttachment`, `TestContainerRemoteManifestReaper_KeysOnTheTombstone/a live manifest at the tombstoned one's digest is untouched`, `TestContainerRemoteImageReaper_KeysOnTheRowItIsHanded/a live image at the tombstoned one's name is untouched`, `.../the reap reaches no other namespace`. `container_remote_blobs`, `container_remote_tags` and `container_remote_manifest_relationships` are uncovered: all three landed after the plan sized its steps, and the plan scopes them out of this one.
AC-20	After a subtree is reaped every blob it referenced has zero attachments, counted per `sha256`	`TestContainerRemoteManifestReaper_Reap/removes the tombstoned manifest and its attachment`, `TestContainerRemoteImageReaper_Reap/reaps the image with the manifests beneath it, in both states`, `TestContainerRemoteImageReaper_BoundsEachChunk`.
AC-21	`DELETE` without `destructive`, or with a non-boolean value, returns `400`	Not this step. Step 18.
AC-22	`destructive=false` on an empty repository returns `204` and frees the name	Not this step. Step 18.
AC-23	`destructive=false` on a non-empty repository returns `409`, contents intact	Not this step. Step 18.
AC-24	That `409` names the remedy the caller has; `code` is `conflict`	Not this step. Step 18.
AC-25	`destructive=true` on a non-empty repository: `202`, tombstone, transactional enqueue	Not this step. Step 18.
AC-26	`destructive=true` on an empty repository returns `204`, no tombstone and no job	Not this step. Step 18.
AC-27	Every client route naming the tombstoned repository returns `404`, per format	Not this step. Steps 3 and 18.
AC-28	A create at a tombstoned name returns `409` until the finalizer runs	Not this step. Step 18.
AC-29	The finalizer cannot succeed early with an artifact left in place	Not this step. Step 14.
AC-30	A finalizer reaching an empty subtree removes the repository row and its links	Not this step. Step 14.
AC-31	A tombstoned repository with no live purge job is re-queued within one `sweep_interval`	Not this step. Step 16.
AC-32	The sweep reads oldest-first, verified by seeding and by `EXPLAIN`	Not this step. Steps 5 and 16.
AC-33	A tombstoned `npm_packages` row whose versions are reaped is enumerated and re-queued	Not this step. Steps 7 and 16.
AC-34	A repository that already has a live purge job is not re-queued	Not this step. Step 16.
AC-35	A prior purge job in a terminal state is re-queued, pinning `UniqueByArgsWhileLive()`	Not this step. Step 16.
AC-36	A permanently failing purge stays counted and re-queued at the bounded cadence	Not this step. Steps 16 and 17b.
AC-37	The sweep registers a worker for its periodic kind	Not this step. Step 16.
AC-38	A namespace that does not serve writes has no tombstone enqueued, and is re-queued once the condition clears	Not this step. Steps 15a and 16.
AC-39	`ContainerRepositoryStore.FindByNameInNamespace` returns `ErrNotFound` for both container formats	Not this step. Step 3.
AC-40	`RepositoryStore.FindByID` returns `ErrNotFound` for a tombstoned repository	Not this step. Step 3.
AC-41	`RepositoryStore.Update` returns `ErrNotFound` for a tombstoned repository	Not this step. Step 3.
AC-42	`Delete`, the three counter writes and `Missing` still act on a tombstoned row	Not this step. Step 3.
AC-43	`parentRepositoryIsActive` refuses a parent tombstoned mid-transaction	Not this step. Step 4.
AC-44	The four `container_images` gates hide a tombstoned image, asserted per statement	Not this step. S17 Phase 4's Step 6.
AC-45	Every `/v2/` route resolving a tombstoned image answers `404 NAME_UNKNOWN`	Not this step. S17 Phase 4's Step 6.
AC-46	The blob mount route answers `202` when every source image is tombstoned	Not this step. S17 Phase 4's Step 6.
AC-47	A tombstoned image named in `from=` still mounts and answers `201`	Not this step. S17 Phase 4's Step 6.
AC-48	The mount destination is not a `404` case	Not this step. S17 Phase 4's Step 6.
AC-49	`UpsertContainerImage` does not resurrect a tombstoned image, asserted on the id	Not this step. S17 Phase 4's Step 6.
AC-50	`ContainerImageStore.Delete` still acts on a tombstoned row	Not this step. S17 Phase 4's Step 6.
AC-51	The destructive repository `DELETE` emits no counter delta at the tombstone write	Not this step. Step 19.
AC-52	`Δsize` and `Δcomponents` fire at the purger, one format-dispatched site	Not this step. Step 19. `ReapTotals.Components` is asserted here as the manifest count that site will read; no positive value is asserted for `ReapTotals.SizeBytes`, for the reason in the suite's header comment.
AC-53	Every emit site commits its source rows before issuing the delta	Not this step. Step 19.
AC-54	A purge chunk emits from its committed affected-row aggregate, not its discovery inventory	Not this step. Step 19. This suite asserts the affected-row half of the same rule: every totals assertion reads the rows the chunk removed.
AC-55	A raw foreign-key cascade emits nothing and is not used	Held here as a schema fact rather than a test: no container-remote child foreign key declares `ON DELETE CASCADE`, so `TestContainerRemoteImageReaper_Reap/reaps the image with the manifests beneath it, in both states` fails unless something walks the key. Deleter observation is Steps 13 and 14.
AC-56	The two discovery-index migrations apply and roll back cleanly	Not this step. Step 1.
AC-57	No migration adds or drops a column, and no `CHECK` pins `soft_deleted_at`	Not this step. Step 1. This step adds no migration.
AC-58	`unique_container_images_ns_id_cr_id_name` is partial and no non-partial form survives	Not this step. S17 Phase 4's Step 5.
AC-59	Config load accepts, rejects and bounds the two lifecycle values	Not this step. Step 2.

Error cases

#	Condition	Tests
E-1	`DELETE` omits `destructive`: `400`, nothing written	Not this step. Step 18.
E-2	`destructive` carries a value other than `true` or `false`: `400`	Not this step. Step 18.
E-3	`destructive=false` on a non-empty repository: `409`, message per case	Not this step. Step 18.
E-4	An artifact published between the `DELETE` arriving and its `DELETE FROM repositories`	Not this step. Step 18.
E-5	`DELETE` on a missing or already-tombstoned repository: `404`	Not this step. Step 18.
E-6	Create at a name held by a tombstoned repository: `409`	Not this step. Step 18.
E-7	Any client route naming a tombstoned repository: `404` in its own vocabulary	Not this step. Steps 3 and 18.
E-8	A blob mount whose `from=` names a tombstoned repository: `202`	Not this step. Step 18.
E-9	Purge job: `23503` on a parent delete aborts the transaction and fails the job	Reaper half covered: `TestContainerRemoteManifestReaper_SparesStillReferencedAttachment` is what keeps a still-referenced attachment off that path. Job-level classification is Step 15's. The three unreaped container-remote child tables leave a real `23503` route open; see AC-19.
E-10	Purge job: `ContainerManifestDeleter` refuses a still-referenced manifest	Not this step. Step 13.
E-11	Purge job: the finalizer aborts because an artifact remains	Not this step. Step 14.
E-12	Purge job: repository already hard-deleted when the job runs, reported as success	Reaper half covered: `TestContainerRemoteManifestReaper_Reap/re-reaping an already reaped manifest is a no-op that reports success`, `TestContainerRemoteImageReaper_Reap/re-reaping an already reaped image is a no-op that reports success`.
E-13	Purge job: retention window changed between enqueue and run	Not this step. `Reaper.Reap` takes no window, so the run-time predicate is Steps 5 and 15.
E-14	Sweep: enumeration query fails, the tick errors and River retries	Not this step. Step 16.
E-15	Sweep: the interval elapses before enumeration completes	Not this step. Step 16.
E-16	Emit API unavailable at a purge or tombstone site	Not this step. Step 19.
E-17	Parent gate: parent tombstoned between the gate and the child `INSERT`	Not this step. Step 4.
E-18	A tombstoned artifact's coordinate reoccupied by a republished row before the reap	`TestContainerRemoteManifestReaper_KeysOnTheTombstone/a live manifest at the tombstoned one's digest is untouched`, `TestContainerRemoteImageReaper_KeysOnTheRowItIsHanded/a live image at the tombstoned one's name is untouched`.

Security considerations

#	Concern	Tests
S-1	Authorization unchanged; the purger holds no caller identity, its authority is the tombstone	Held as the tombstone-keying property, at the level each reaper's authority sits. `TestContainerRemoteManifestReaper_KeysOnTheTombstone/a live manifest handed to the reaper is refused` pins the manifest reaper against its own row's state. The image reaper is state-blind on the row it is handed, and its authority is the tombstone on the repository above that row, stated as a precondition on `Reap`. `TestContainerRemoteImageReaper_KeysOnTheRowItIsHanded/a live image handed to the reaper is reaped` pins that behavior. Neither reaper holds a caller identity, and while `ScanTombstonedRepositories` stays the only source of a dispatched `TombstoneRow`, no row with no tombstone above it is dispatched.
S-2	`destructive` is a confirmation, not an authorization control	Not this step. Step 18.
S-3	Deletion is irreversible in closed beta and the API must say so	Not this step. Step 18 and the OpenAPI text.
S-4	No purger input is user-controlled; the purge job carries UUID identifiers only	`TestContainerRemoteReapers_ArgumentGuards`, twelve subtests across both reapers. Every reap input is a UUID, an `int64` attachment id or an `int` limit, so no format-layer name, version string or digest reaches a statement as text.
S-5	No injection surface is added; all purger SQL is in `internal/datastore` and parameterized	No test. The statements live in `internal/datastore/lifecycle_reap_container_remote.go` and `blob_storage_attachments.go` by construction, and `TestEveryStatementIsInstrumented` pins how each reaches the driver. Parameterization is reviewed rather than asserted, as it is for `DeleteIfUnreferenced` today.
S-6	Error messages must not become an existence oracle	Not this step for the client-facing half: the reap answers no caller, and Steps 3 and 18 own the `404`. What this suite pins is the identifier-free wrap rule the same concern rests on, at every one of the six wrap sites in `internal/datastore/lifecycle_reap_container_remote.go`. The two chunk deletes are covered by `TestContainerRemoteManifestReaper_Reap/a statement failure wraps the cause and names no identifier` and its image-level counterpart in `TestContainerRemoteImageReaper_Reap`. The attachment free and the image delete are covered by the two `container_remote_blobs` abort subtests in those same two tests. The digest conversion, the image delete and its row count are covered by `internal/datastore/lifecycle_reap_container_remote_test.go`, which drives them against a fake handle because no live database reaches them.
S-7	Purge failures are operationally visible rather than silent	Not this step. Steps 16, 17a and 17b own the signals. This suite's contribution is that a reap never reports work it did not do: every totals assertion reads the rows the chunk removed, and a failed chunk reports `ReapTotals{}`.

e2e scenario catalogs

No scenario is added or affected. docs/testing/e2e/oci.md:28 and docs/testing/e2e/docker.md:28 both list "Virtual and remote (proxy/cache) repositories" under Out of scope until the capability ships. This step reaps only remote-cache tables.

Diff size

The diff passes the 500 reviewable-LOC line. Split by file group, from git diff --numstat origin/main...HEAD at 7b07b179f, whose merge base is 3f0d92e65:

Group	Files	Insertions	Deletions
Production	`lifecycle_reap_container_remote.go`, `blob_storage_attachments.go`, `query_names.go`, `container_manifest_deleter.go`	898	12
Test	`lifecycle_reap_container_remote_integration_test.go`, `lifecycle_reap_container_remote_explain_integration_test.go`, `lifecycle_reap_container_remote_test.go`, `blob_storage_attachments_integration_test.go`, `blob_storage_attachments_stmt_test.go`, `blob_storage_attachments_test.go`, `queries_test.go`	2266	15
Spec	`docs/specs/S20-a-lifecycle-closed-beta.md`	7	3
Plan	`docs/plans/2026-08-11-s20a-lifecycle-closed-beta.md`	21	6
Total	13 files	3192	36

The command is named rather than a base and head SHA pair, because a fixed pair goes stale on every rebase and this table had gone stale that way twice.

The 1531-line integration test is the bulk. The figures include two review passes, together 1851 insertions and 323 deletions across 13 files on top of a4fa452ef, the commit that first implemented the reap. The first pass added the reap fixture's repository tombstone with the repository-delete subtest, the narrow attachment statement's structural pins, a direct suite for the attachment guard with its argument guards, the 187-line partition-pruning suite, both spec amendments, and comment corrections across both production files. The second routed both parent deletes through mapReapParentDeleteError with a container_remote_tags abort subtest, added the connection-pool guard and moved the suite onto one transaction per chunk, retargeted the page ceiling, and rewrote the mirrored-suite header as a diff against the three merged reap suites with the shared-attachment coverage it found missing.

A split does not help here. The reap and its suite are one deliverable, and the acceptance criteria are asserted by that suite. The plan's Size forecast and the working ceiling section adopts a ~900 added-LOC working ceiling for these steps, and forecasts this one at ~570. This diff is over both figures.

Merge-order notes

1. The const-block realignment: steps 8 and 9 merged, this MR rebased onto both

This MR, S20-A step 8 and S20-A step 9 each add one long constant to the blob_storage_attachments group in internal/datastore/query_names.go. gofmt realigns the whole group to a different width for each one, so the three conflict in that group whatever order they merge in.

Both siblings have merged and this branch sits on both. Step 8 is !1705 (merged), merged as bb1032b9. Step 9 is !1702 (merged), merged as f7923980. Both merge commits are ancestors of this branch's head, so neither rebase is owed.

internal/datastore/queries_test.go behaves differently. This MR realigns rawSQLTimedFunctions, because DeleteIfUnreferencedByContainerRemoteManifest is the longest entry. That realignment is still live against one open sibling; see note 4.

Treat each rebase as deliberate rather than mechanical. The conflict is an alignment width across a whole block, and a resolution that takes one side whole drops the other side's constant. The cause is easy to miss: each step adds one long constant, gofmt realigns the block, and no per-step review shows the realignment.

2. The query-name budget

The name label budget in internal/metrics/cardinality.go:58 is 400. An earlier version of this note said 350 and treated S20-A step 6's raise as a precondition; both were wrong, and the raise merged.

Counted at this MR's head: 372 names, 360 in internal/datastore/query_names.go and 12 in internal/storage/queries.go, so 28 under the budget. On origin/main the same count is 368, so this MR's arm is the 4 names below. TestNameBudget_CoversEveryDeclaredQueryName stays green in every merge order, and this MR's position in that order does not matter to it.

This MR's arm is 4 names:

blob_storage_attachments_delete_if_unreferenced_by_container_remote_manifest
container_remote_images_delete_reap
container_remote_manifests_delete_reap_tombstoned
container_remote_manifests_delete_reap_under_image

The three reap statements carry a reap token, so database_query_duration_seconds{name=~".*reap.*"} reaches this family as it reaches the merged Maven and npm ones. The attachment guard does not, and blob_storage_attachments.go records why a guard named for its one correlation table is the exception. No production code constructs either reaper, so none of the four series has been emitted and the naming costs no dashboard.

3. What S20-A step 9 settled for this MR

Step 9 is !1702 (merged), merged, and in this branch's merge base. Two of this MR's open ends closed there.

The reap-page ceiling. MaxLifecycleReapPageSize is on main at internal/datastore/lifecycle_reap_maven.go:41. This branch now sits on it: 730ef83da clamped the page at both ends against the discovery-scan ceiling, and guardContainerRemoteReap rejects limit < 1 and limit > MaxLifecycleReapPageSize. Both constants are 1000, so the bound a caller sees did not move when the guard was retargeted. The future-tense pointer that stood in for the retarget is gone with it, and the sentinel's message names no constant, which is the form every other reap arm uses. No caller of either constructor exists, so the window closed before anything could exercise it.

What limit bounds. The merged Reaper interface states "At most limit rows are deleted per call", and neither reaper here does that. limit bounds the container_remote_manifests rows one call removes, with the attachment freed for each of them and the image row on top of that bound, so ReapTotals.Rows runs past limit. Both Reap docs in this MR state that reading. !1702 (merged) amended the interface obligation to the same reading, with a 2*limit ceiling on one call, at internal/datastore/lifecycle_scan.go:162. That amendment is on main, so the interface and these two implementations now agree.

4. Open siblings, and the two conflicts a resolver will meet

Computed against this MR's head and the current origin/main, with git merge-tree --write-tree against each sibling's own head rather than from titles. This MR changes 13 files.

Open MR	Shared paths	Merge result
!1711 (merged) container hosted reap, step 13	the spec, `internal/datastore/container_manifest_deleter.go`, `internal/datastore/query_names.go`	conflict in `container_manifest_deleter.go`, one hunk. The spec and `query_names.go` auto-merge
!1754 (merged) S17 Phase 4, 38/38	`internal/datastore/container_manifest_deleter.go`, `internal/datastore/queries_test.go`, `internal/datastore/query_names.go`	conflict in `queries_test.go`. The other two auto-merge
!1746 (merged) npm remote cache reap, step 10	the spec, `internal/datastore/query_names.go`	clean
!1787 (merged) plan Status batch	the plan	clean, because it edits row 7 only

No merge order is stated, and none is needed: the dependency matrix calls steps 8, 9 and 12 mutually independent, and Step 14 already gates on steps 9 and 12 together. What a resolver needs instead is the rule note 1 states for the const block, applied to each conflicted file.

container_manifest_deleter.go against !1711 (merged). This MR rewrites the blobSHA256 doc paragraph; !1711 (merged) inserts validateCascadeWriteHandle and its sentinel immediately above it, with no unchanged line between the two regions. The loss is asymmetric, so a resolver cannot read it off a green build. Taking this MR's side whole deletes func validateCascadeWriteHandle and leaves its call site, which fails to compile loudly. Taking !1711 (merged)'s side whole deletes only this MR's doc paragraph, silently. Keep both.

queries_test.go against !1754 (merged). Same shape as note 1's const block: this MR realigns rawSQLTimedFunctions and !1754 (merged) adds three entries to it. Neither realigns the other's rows, so take both sides' entries and let gofmt set the width. This MR's container_manifest_deleter.go merges clean against !1754 (merged), so a resolver should not look for a conflict there.

One name collision that no merge resolves. This MR declared func reapManifestRow in internal/datastore/lifecycle_reap_container_remote_integration_test.go, and !1711 (merged) declares type reapManifestRow struct in internal/datastore/lifecycle_reap_container_integration_test.go. Both are package datastore under //go:build integration, so git merges them cleanly and the tagged build then stops at a redeclaration. No CI lint job passes that build tag, so it would surface only in the integration test job of whichever MR merged second. This MR renamed its helper to reapContainerRemoteManifestRow, so the collision is gone whatever order the two take. That follows what this branch already did for drainReap, which collided with the one !1705 (merged) merged.

Notes for the reviewer

Four sibling spec claims are left standing, deliberately

This MR corrects three spec claims that the reap contradicts, and leaves four of the same family untouched rather than sweeping them. Naming them here so a reader does not find that one was fixed and its neighbour was not.

Acceptance criterion 18 seeds by "writing soft_deleted_at directly where no writer exists yet" over seven tables, and a writer exists for three of them: npm_remote_packages, npm_remote_versions and container_remote_images all have a merged evictor that sets the column, each without a production caller.
The Resolutions entry for #470 (closed) says "npm is still the only format that writes the column". Maven and container both have merged writers.
The reachability table lists container_remote_images, container_remote_manifests and container_remote_blobs under Reachable by tombstone scan. Only container_remote_manifests carries a tombstone-discovery index, which the Data Model section states two sections earlier.
The hosted-container section says "Maven cannot reach this shape until a writer exists". The two Maven writers exist and have no caller, which is the distinction this MR's own Data Model correction now draws.

All four are pre-existing and none is in this MR's diff. Correcting them reaches four sections this step does not otherwise touch, which is the same scope line already drawn for the two GA-scoped occurrences of the reap claim. No work item covers them today; #534 is the nearest existing home, and its stated scope is comment and doc corrections in other files rather than spec criteria, so adopting these would widen it.

No e2e scenario is added or affected

No scenario in docs/testing/ reaches the purger. Nothing dispatches a Reaper until Step 16, which is the step after which a tombstone on a running server is actually reaped, so there is no scenario to move for a change to a reaper's chunk boundary. The reap's own behavior is covered by internal/datastore/lifecycle_reap_container_remote_integration_test.go.

The plan Status row

Step 12's row in docs/plans/2026-08-11-s20a-lifecycle-closed-beta.md carries !1704, filled in this MR's own change set, as guardrail 4 asks.

An earlier revision of this section said the row stayed empty on purpose, for a separate MR to fill every row at once. That reading does not hold. Across this plan and the S22 plan, 17 step MRs are merged, and 16 of them filled their own row. S20-A step 7 (!1700 (merged)) is the one exception. No batch-fill MR exists for either plan. The batch mechanism is real practice elsewhere, and it has never touched S20-A or S22.

Eight files sit outside the plan's `Files` list

Step 12's Files list names lifecycle_reap_container_remote.go, blob_storage_attachments.go and lifecycle_reap_container_remote_integration_test.go. The other eight files in this diff are:

internal/datastore/query_names.go and internal/datastore/queries_test.go, compile-forced by merged guards: TestNameBudget_CoversEveryDeclaredQueryName reads the query-name catalog, and TestEveryStatementIsInstrumented reads the rawSQLTimedFunctions pin. Merged step 5 set the precedent for this pair, because its squash 0630023a added to query_names.go and its own Files list does not name that file.
internal/datastore/blob_storage_attachments_stmt_test.go, internal/datastore/blob_storage_attachments_integration_test.go and internal/datastore/blob_storage_attachments_test.go, which carry the structural pins, the direct suite and the argument-guard cases for DeleteIfUnreferencedByContainerRemote. They sit beside the tests of DeleteIfUnreferenced, the method this one narrows, because that is where the pins they mirror live.
internal/datastore/container_manifest_deleter.go, which holds blobSHA256. Its doc named one caller's column while the function has six callers, and this diff adds the sixth.
internal/datastore/lifecycle_reap_container_remote_explain_integration_test.go, the partition-pruning suite described under The reap's four statements carry a partition-pruning suite. A file of its own rather than an addition to lifecycle_explain_integration_test.go: the package names its ten other explain suites one per production file or family, that file is the discovery half of this seam rather than the reap half, and steps 6 and 7 are both appending to it.
docs/specs/S20-a-lifecycle-closed-beta.md, which carries the two spec amendments described above.

One spec criterion is deferred

container_remote_blobs, container_remote_tags and container_remote_manifest_relationships are out of scope for this step. All three tables arrived after this plan sized its steps. The plan defers them to a follow-on MR against the same reaper, in its Research Findings and Dependencies sections. That follow-on is tracked as #690, and the plan's Dependencies bullet now names the merge order it owes: it must merge before Step 16's MR opens.

The consequence is stated rather than left for a reader to find. No container-remote child foreign key declares ON DELETE CASCADE, and the keys are not one per table. All three tables carry a key onto container_remote_images, and container_remote_tags and container_remote_manifest_relationships carry keys onto container_remote_manifests as well: one for a tag's manifest, two for a relationship's parent and child. A surviving row in any of the three therefore aborts the image-row delete, and a surviving tag or relationship row also aborts the child delete of the manifest it pins, with SQLSTATE 23503 either way. Three shapes are therefore not reapable until these reapers cover those three tables:

An image that cached anything by tag.
An index that recorded its children.
An image that holds cached layers.

The file-head comment in lifecycle_reap_container_remote.go records the same limitation, and the spec-coverage table marks it at AC-19 and E-9.

The short-chunk comment is corrected, with no behaviour change

The comment above the image reap's chunk-length check claimed too much. It read a count below limit as proof the subtree had drained, and said nothing was left for the image row's foreign key to catch. That does not hold under READ COMMITTED: the chunk subquery chooses its ids from the statement's own snapshot, the outer delete ignores each id another transaction has deleted and committed, and nothing tops the chunk back up to limit. A short chunk is therefore the signal that the image row is worth attempting, not proof the attempt succeeds.

The comment now says that, and says what the attempt costs when it fails: SQLSTATE 23503 aborts the whole chunk transaction, so the chunk's manifest deletes and the attachment deletes that followed them roll back with it, and a retry repeats that work rather than continuing from it. No NOT EXISTS guard is added and no statement text changes. The spec's error table already accepts an aborted transaction and a re-queued job as the outcome of a 23503 on a parent delete, so a guard would narrow a statement the spec expects to abort, and it would report success where the statement aborts today — on a destructive path whose only caller does not exist yet. Its benefit is also small next to the deferral above, because container_remote_blobs aborts the same statement for any image that served a blob request. The merged Reaper contract in internal/datastore/lifecycle_scan.go already records the same divergence between what a projection found and what the delete reached, and the corrected comment cites it rather than restating it.

The reap's four statements carry a partition-pruning suite

internal/datastore/lifecycle_reap_container_remote_explain_integration_test.go asserts one hash partition per table, per statement. Each of the four doc comments claims that pruning and nothing checked it: a dropped namespace_id in the chunk subquery, or a dropped sha256 in the attachment delete, leaves every behavioural test green and starts reading 64 partitions per reaped row.

deleteTombstonedContainerRemoteManifestStmt: one container_remote_manifests partition.
deleteContainerRemoteManifestsUnderImageStmt: one container_remote_manifests partition, which covers the outer delete and the chunk subquery together, and one container_remote_images partition for the EXISTS gate.
deleteContainerRemoteImageStmt: one container_remote_images partition.
deleteAttachmentUnreferencedByContainerRemoteStmt: one blob_storage_attachments partition, pruned on sha256 rather than on the id, and one container_remote_manifests partition for the correlated NOT EXISTS arm.

It pins the pruning half of the plans recorded under Database Review Evidence, which stay the one-time measurement of cost and buffers.

Two constraints, both stated in the file header. Plain EXPLAIN, never EXPLAIN ANALYZE: ANALYZE executes the statement it plans, and this is the package's first explain suite over a write that removes rows. And no access-path assertion: the chunk subquery is answered by index_container_remote_manifests_on_ns_id_image_id_last_dl_at or by a LIMIT-stopped Seq Scan depending on how much of the namespace sits under the one image, both of them correct, so an Index Cond assertion would pin the fixture and could fail against a correct statement. The suite adds no rule to the plan's Testing Strategy: it is coverage in this MR rather than a new standing requirement on the other reap steps.

One comment diverges from the plan's wording, deliberately

The plan's step 12 Scope line says the two tables' NOT NULL blob_storage_attachment_id "puts them under ADR-007's same-transaction attachment rule". ADR-007 carries no nullability qualifier, and it binds every client of blob_storage_attachments. NOT NULL is what makes the ordering forced here, not what puts the table under the rule. DeleteIfUnreferencedByContainerRemote's doc comment carries the corrected wording and the plan's Scope line keeps its own, so this is a deliberate divergence rather than a transcription error.

ADR freshness

scripts/adr-freshness.sh exits 1, so treat the local mirror as stale rather than current. It is stale on one upstream commit, and that commit touches ADR-020 only. No file in this diff cites ADR-020. Internal ADRs went uncounted, because the internal handbook was not reachable with this token.

Database Review Evidence

Query mode only. The diff adds no migration, so migration mode did not run. git diff --name-only origin/main...HEAD -- internal/datastore/migrations/ is empty.

Queries

Note

Plans are from EXPLAIN (ANALYZE, BUFFERS) against an ephemeral PostgreSQL 17.10 container (matching GL_PG_CURR_VERSION: "17" from .gitlab-ci-other-versions.yml), with synthesized seed data rolled back per query and the container torn down at the end of the run. Numbers reflect moderate cardinality and do not capture production-scale effects. See Database review evidence for seed sizing, methodology, and the anomalies the skill flags. Expand each row's details for the seed shape, rendered SQL, bound args, and raw plan.

Method	Plan node	Index	Rows (plan / actual)	Cost	Time	Buffers (hit / read)	Partitions
`datastore.BlobStorageAttachmentStore.DeleteIfUnreferencedByContainerRemote`	Delete (Nested Loop Anti Join)	`n/a`	0 / 0	5.65	0.802ms	63 / 0	1/64, 1/64
`datastore.ContainerRemoteImageReaper.Reap`	Delete	`n/a`	10 / 10	5.39	0.184ms	53 / 0	1/64, 1/64
`datastore.ContainerRemoteManifestReaper.Reap`	Delete	`n/a`	1 / 1	3.75	0.146ms	24 / 0	1/64
`datastore.deleteTombstonedContainerRemoteImage`	Delete	`n/a`	0 / 0	1.75	0.332ms	24 / 0	1/64

Every statement prunes to one partition of every partitioned table it reads. No fan-out, so no partition warning.

The Index column is n/a on all four rows for one reason. All four statements are write targets, and the recipe seeds a write target with 1 target row plus 49 siblings. Each target partition then holds about 50 rows, which is 2 to 3 heap pages, and a Seq Scan of that is cheaper than any index. The 50-row figure is below the 5000-row threshold at which the recipe treats a Seq Scan as a finding, so the table reports no anomaly.

That leaves the index question open, so a second pass answers it. The table below repeats each statement against 5000 rows in the one partition. It is extra evidence rather than part of the recipe.

Method	Plan node	Index	Rows (plan / actual)	Cost	Time	Buffers (hit / read)	Partitions
`datastore.BlobStorageAttachmentStore.DeleteIfUnreferencedByContainerRemote`	Delete (Nested Loop Anti Join)	`index_container_remote_manifests_on_ns_id_bsa_id`	0 / 0	11.21	0.291ms	63 / 0	1/64, 1/64
`datastore.ContainerRemoteImageReaper.Reap` (selective image)	Delete	`index_container_remote_manifests_on_ns_id_image_id_last_dl_at`, `pk_container_remote_manifests`	10 / 10	107.33	0.214ms	83 / 0	1/64, 1/64
`datastore.ContainerRemoteImageReaper.Reap` (one image holds every row)	Delete	`pk_container_remote_manifests`, `pk_container_remote_images`	10 / 10	88.07	0.416ms	83 / 0	1/64, 1/64
`datastore.ContainerRemoteManifestReaper.Reap`	Delete	`index_container_remote_manifests_on_ns_id_soft_deleted_at_id`	1 / 1	8.15	0.215ms	23 / 0	1/64
`datastore.deleteTombstonedContainerRemoteImage`	Delete	`pk_container_remote_images`	0 / 0	8.30	0.122ms	26 / 0	1/64

Each statement reaches the index its doc comment names. The manifest reap lands on the partial tombstone index with all three conjuncts as Index Cond. The chunk subquery lands on index_container_remote_manifests_on_ns_id_image_id_last_dl_at with both leading columns as Index Cond, which is the claim in deleteContainerRemoteManifestsUnderImageStmt. The NOT EXISTS arm lands on index_container_remote_manifests_on_ns_id_bsa_id.

Query notes:

datastore.ContainerRemoteImageReaper.Reap: limit is clamped at both ends. guardContainerRemoteReap rejects limit < 1 and limit > MaxLifecycleReapPageSize, so the statement renders at most 1000 as LIMIT $4. One chunk's transaction size and its RETURNING result set are therefore bounded by the server as well as by the caller. The figures below were measured at limit = 10, so they do not describe the ceiling.
All four statements: referential-integrity trigger time dominates every measured execution, and the chunk size multiplies it. The image reap spends 170.8ms of its 171.3ms execution in three triggers at 10 calls each. The attachment delete spends 62.2ms of its 64.2ms in ten triggers at 1 call each, one per table that carries a foreign key into blob_storage_attachments. Read the second figure against the doc comment on DeleteIfUnreferencedByContainerRemote. That comment claims the one-arm statement is cheaper than the ten-arm one, and it says so about plan-time work, which the numbers support (planning is 3.8ms with 201 buffer hits). PostgreSQL still verifies all ten foreign keys on the row it removes, so the narrow correlation does not reduce the referential check itself.
Trigger figures above are stats-sensitive, and the recipe leaves them so. container_remote_tags, container_remote_manifest_relationships and container_remote_blobs hold no rows and sit outside the seed set, so the recipe never runs ANALYZE on them. A probe run that adds them to the ANALYZE list drops the image reap's trigger total from 170.8ms to 37.5ms, which is still about 3.7ms for each row the chunk removes.
datastore.ContainerRemoteImageReaper.Reap: the chunk subquery's plan depends on how selective the image predicate is. Where one image holds all 5000 rows of the partition, the planner takes a Seq Scan and the LIMIT stops it after 10 rows (0.010ms actual against a 211.00 full-scan estimate). Where 250 images share the 5000 rows, it takes the index. Production matches the second shape, because a partition holds many namespaces and many images.
containerRemoteImageExists renders as an InitPlan under a One-Time Filter, so PostgreSQL evaluates the EXISTS gate once for each statement rather than once for each candidate row.

datastore.BlobStorageAttachmentStore.DeleteIfUnreferencedByContainerRemote

Summary: The plan matches the method's intent. A Nested Loop Anti Join carries the correlated NOT EXISTS, and both correlated columns prune their tables to one partition each, blob_storage_attachments_p00 and container_remote_manifests_p26. At 5000 referencing rows the inner side takes index_container_remote_manifests_on_ns_id_bsa_id with both columns as Index Cond. The statement itself carries no anomaly. Its execution time is set by the ten foreign-key triggers, which is a property of the table rather than of this statement.

Seed shape: namespaces=1, repositories=1, container_remote_repositories=1, container_remote_images=50, blob_storage_blobs=51, blob_storage_attachments=51, container_remote_manifests=50

blob_storage_attachments and blob_storage_blobs are PARTITION BY HASH (sha256), so the 51 digests come from satisfies_hash_partition(..., 64, 0, ...) and land in one partition. Attachment 51 is the target, and no manifest references it. The other 50 attachments each carry one referencing manifest row.

Rendered SQL (raw const, not a jet chain, so nothing renders it):

DELETE FROM blob_storage_attachments bsa
	WHERE bsa.namespace_id = $1 AND bsa.id = $2 AND bsa.sha256 = $3
	AND NOT EXISTS (
		SELECT 1 FROM container_remote_manifests ref
		WHERE ref.namespace_id = bsa.namespace_id AND ref.blob_storage_attachment_id = bsa.id
	)

Bound args: [cbf6c68c-d5cb-4e62-850b-631a6e72cdd4, 306, \x0000000000000000000000000000000000000000000000000000000000000ba5]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Delete on blob_storage_attachments bsa  (cost=0.00..5.65 rows=0 width=0) (actual time=0.800..0.802 rows=0 loops=1)
   Delete on blob_storage_attachments_p00 bsa_1
   Buffers: shared hit=63
   ->  Nested Loop Anti Join  (cost=0.00..5.65 rows=1 width=20) (actual time=0.058..0.062 rows=1 loops=1)
         Buffers: shared hit=4
         ->  Seq Scan on blob_storage_attachments_p00 bsa_1  (cost=0.00..2.89 rows=1 width=34) (actual time=0.021..0.023 rows=1 loops=1)
               Filter: ((namespace_id = 'cbf6c68c-d5cb-4e62-850b-631a6e72cdd4'::uuid) AND (id = '306'::bigint) AND (sha256 = '\x0000000000000000000000000000000000000000000000000000000000000ba5'::bytea))
               Rows Removed by Filter: 50
               Buffers: shared hit=2
         ->  Seq Scan on container_remote_manifests_p26 ref  (cost=0.00..2.75 rows=1 width=34) (actual time=0.035..0.036 rows=0 loops=1)
               Filter: ((namespace_id = 'cbf6c68c-d5cb-4e62-850b-631a6e72cdd4'::uuid) AND (blob_storage_attachment_id = '306'::bigint))
               Rows Removed by Filter: 50
               Buffers: shared hit=2
 Planning:
   Buffers: shared hit=201
 Planning Time: 3.838 ms
 Trigger for constraint container_blobs_blob_storage_attachment_id_namespace_id_bl_fkey on blob_storage_attachments_p00: time=8.019 calls=1
 Trigger for constraint container_manifests_blob_storage_attachment_id_namespace_i_fkey on blob_storage_attachments_p00: time=6.674 calls=1
 Trigger for constraint npm_files_blob_storage_attachment_id_namespace_id_blob_sha_fkey on blob_storage_attachments_p00: time=3.350 calls=1
 Trigger for constraint npm_metadata_files_blob_storage_attachment_id_namespace_id_fkey on blob_storage_attachments_p00: time=14.368 calls=1
 Trigger for constraint maven_files_blob_storage_attachment_id_namespace_id_blob_s_fkey on blob_storage_attachments_p00: time=4.736 calls=1
 Trigger for constraint npm_remote_metadata_files_blob_storage_attachment_id_names_fkey on blob_storage_attachments_p00: time=5.672 calls=1
 Trigger for constraint npm_remote_files_blob_storage_attachment_id_namespace_id_b_fkey on blob_storage_attachments_p00: time=6.352 calls=1
 Trigger for constraint maven_remote_files_blob_storage_attachment_id_namespace_id_fkey on blob_storage_attachments_p00: time=6.540 calls=1
 Trigger for constraint container_remote_manifests_blob_storage_attachment_id_name_fkey on blob_storage_attachments_p00: time=0.753 calls=1
 Trigger for constraint container_remote_blobs_blob_storage_attachment_id_namespac_fkey on blob_storage_attachments_p00: time=5.657 calls=1
 Execution Time: 64.242 ms

Timings: planning 3.838ms, execution 64.242ms, total 68.080ms.

Extra pass, 5000 referencing rows (container_remote_manifests=5000 in the one partition):

 Delete on blob_storage_attachments bsa  (cost=0.28..11.21 rows=0 width=0) (actual time=0.290..0.291 rows=0 loops=1)
   Delete on blob_storage_attachments_p00 bsa_1
   Buffers: shared hit=63
   ->  Nested Loop Anti Join  (cost=0.28..11.21 rows=1 width=20) (actual time=0.030..0.031 rows=1 loops=1)
         Buffers: shared hit=4
         ->  Seq Scan on blob_storage_attachments_p00 bsa_1  (cost=0.00..2.89 rows=1 width=34) (actual time=0.017..0.017 rows=1 loops=1)
               Filter: ((namespace_id = '27b4e8de-e36f-42b4-968f-0fcc0d63c28f'::uuid) AND (id = '510'::bigint) AND (sha256 = '\x0000000000000000000000000000000000000000000000000000000000000ba5'::bytea))
               Rows Removed by Filter: 50
               Buffers: shared hit=2
         ->  Index Scan using container_remote_manifests_p5_namespace_id_blob_storage_at_idx8 on container_remote_manifests_p58 ref  (cost=0.28..8.30 rows=1 width=34) (actual time=0.012..0.012 rows=0 loops=1)
               Index Cond: ((namespace_id = '27b4e8de-e36f-42b4-968f-0fcc0d63c28f'::uuid) AND (blob_storage_attachment_id = '510'::bigint))
               Buffers: shared hit=2
 Planning:
   Buffers: shared hit=171
 Planning Time: 0.989 ms
 Execution Time: 40.634 ms

The partition-local index name container_remote_manifests_p5_namespace_id_blob_storage_at_idx8 is the child of index_container_remote_manifests_on_ns_id_bsa_id. Ten trigger lines are cut from this second block only. They repeat the first block's set.

datastore.ContainerRemoteImageReaper.Reap

Summary: The plan matches the method's intent. The EXISTS gate becomes an InitPlan under a One-Time Filter, so it runs once for each statement. The chunk subquery's LIMIT bounds the delete at 10 rows, and namespace_id prunes container_remote_manifests and container_remote_images to one partition each. Estimates match reality at 10 / 10 rows. One thing stands outside the statement: the chunk subquery's access path depends on how selective the image predicate is. It is in the Query notes above, beside the guard's clamp on limit.

Seed shape: namespaces=1, repositories=1, container_remote_repositories=1, container_remote_images=50, blob_storage_blobs=51, blob_storage_attachments=51, container_remote_manifests=50

All 50 manifest rows hang off image 1, so the subquery has 50 candidates and LIMIT 10 takes 10 of them.

Rendered SQL:

DELETE FROM public.container_remote_manifests
WHERE ((container_remote_manifests.namespace_id = $1::uuid) AND (container_remote_manifests.id IN ((
           SELECT reap_chunk.id AS "reap_chunk.id"
           FROM public.container_remote_manifests AS reap_chunk
           WHERE (reap_chunk.namespace_id = $2::uuid) AND (reap_chunk.container_remote_image_id = $3::uuid)
           LIMIT $4
      )))) AND (EXISTS (
           SELECT container_remote_images.id AS "container_remote_images.id"
           FROM public.container_remote_images
           WHERE (container_remote_images.namespace_id = $5::uuid) AND (container_remote_images.id = $6::uuid)
      ))
RETURNING container_remote_manifests.blob_storage_attachment_id AS "reaped_attachment.id",
          container_remote_manifests.blob_sha256 AS "reaped_attachment.sha256";

Bound args: [31501d91-bce0-45ac-8ef9-40137451ed2e, 31501d91-bce0-45ac-8ef9-40137451ed2e, e97b5d72-ef4f-4d84-bf12-ee540129065e, 10, 31501d91-bce0-45ac-8ef9-40137451ed2e, e97b5d72-ef4f-4d84-bf12-ee540129065e]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Delete on container_remote_manifests  (cost=2.52..5.39 rows=10 width=50) (actual time=0.165..0.184 rows=10 loops=1)
   Delete on container_remote_manifests_p23 container_remote_manifests_1
   Buffers: shared hit=53
   InitPlan 1
     ->  Seq Scan on container_remote_images_p23 container_remote_images  (cost=0.00..1.75 rows=1 width=0) (actual time=0.006..0.006 rows=1 loops=1)
           Filter: ((namespace_id = '31501d91-bce0-45ac-8ef9-40137451ed2e'::uuid) AND (id = 'e97b5d72-ef4f-4d84-bf12-ee540129065e'::uuid))
           Buffers: shared hit=1
   ->  Result  (cost=0.78..3.64 rows=10 width=50) (actual time=0.048..0.058 rows=10 loops=1)
         One-Time Filter: (InitPlan 1).col1
         Buffers: shared hit=5
         ->  Hash Semi Join  (cost=0.78..3.64 rows=10 width=50) (actual time=0.039..0.049 rows=10 loops=1)
               Hash Cond: (container_remote_manifests_1.id = "ANY_subquery"."reap_chunk.id")
               Buffers: shared hit=4
               ->  Seq Scan on container_remote_manifests_p23 container_remote_manifests_1  (cost=0.00..2.62 rows=50 width=26) (actual time=0.005..0.010 rows=50 loops=1)
                     Filter: (namespace_id = '31501d91-bce0-45ac-8ef9-40137451ed2e'::uuid)
                     Buffers: shared hit=2
               ->  Hash  (cost=0.65..0.65 rows=10 width=56) (actual time=0.020..0.021 rows=10 loops=1)
                     Buckets: 1024  Batches: 1  Memory Usage: 9kB
                     Buffers: shared hit=2
                     ->  Subquery Scan on "ANY_subquery"  (cost=0.00..0.65 rows=10 width=56) (actual time=0.008..0.011 rows=10 loops=1)
                           Buffers: shared hit=2
                           ->  Limit  (cost=0.00..0.55 rows=10 width=16) (actual time=0.002..0.004 rows=10 loops=1)
                                 Buffers: shared hit=2
                                 ->  Seq Scan on container_remote_manifests_p23 reap_chunk  (cost=0.00..2.75 rows=50 width=16) (actual time=0.001..0.002 rows=10 loops=1)
                                       Filter: ((namespace_id = '31501d91-bce0-45ac-8ef9-40137451ed2e'::uuid) AND (container_remote_image_id = 'e97b5d72-ef4f-4d84-bf12-ee540129065e'::uuid))
                                       Buffers: shared hit=2
 Planning:
   Buffers: shared hit=196 read=1
 Planning Time: 2.327 ms
 Trigger for constraint container_remote_tags_container_remote_manifest_id_names_fkey23 on container_remote_manifests_p23: time=86.242 calls=10
 Trigger for constraint container_remote_manifest_re_parent_container_remote_man_fkey23 on container_remote_manifests_p23: time=81.611 calls=10
 Trigger for constraint container_remote_manifest_re_child_container_remote_mani_fkey23 on container_remote_manifests_p23: time=2.920 calls=10
 Execution Time: 171.313 ms

Timings: planning 2.327ms, execution 171.313ms, total 173.640ms.

Extra pass, 5000 rows over 250 images at 20 each (the selective, production shape):

 Delete on container_remote_manifests  (cost=28.39..107.33 rows=10 width=50) (actual time=0.185..0.214 rows=10 loops=1)
   Delete on container_remote_manifests_p37 container_remote_manifests_1
   Buffers: shared hit=83
   InitPlan 1
     ->  Seq Scan on container_remote_images_p37 container_remote_images  (cost=0.00..7.75 rows=1 width=0) (actual time=0.007..0.007 rows=1 loops=1)
           Filter: ((namespace_id = '17195797-24c3-4531-979f-a3b04b62287f'::uuid) AND (id = 'f851f66d-5d51-412c-a8a6-2e00c51bfb17'::uuid))
           Buffers: shared hit=2
   ->  Result  (cost=20.64..99.58 rows=10 width=50) (actual time=0.061..0.082 rows=10 loops=1)
         One-Time Filter: (InitPlan 1).col1
         Buffers: shared hit=35
         ->  Nested Loop  (cost=20.64..99.58 rows=10 width=50) (actual time=0.052..0.072 rows=10 loops=1)
               Buffers: shared hit=33
               ->  HashAggregate  (cost=20.36..20.46 rows=10 width=56) (actual time=0.031..0.033 rows=10 loops=1)
                     Group Key: "ANY_subquery"."reap_chunk.id"
                     Batches: 1  Memory Usage: 24kB
                     Buffers: shared hit=3
                     ->  Subquery Scan on "ANY_subquery"  (cost=0.28..20.33 rows=10 width=56) (actual time=0.021..0.025 rows=10 loops=1)
                           Buffers: shared hit=3
                           ->  Limit  (cost=0.28..20.23 rows=10 width=16) (actual time=0.016..0.019 rows=10 loops=1)
                                 Buffers: shared hit=3
                                 ->  Index Scan using container_remote_manifests_p_namespace_id_container_remot_idx47 on container_remote_manifests_p37 reap_chunk  (cost=0.28..40.18 rows=20 width=16) (actual time=0.015..0.017 rows=10 loops=1)
                                       Index Cond: ((namespace_id = '17195797-24c3-4531-979f-a3b04b62287f'::uuid) AND (container_remote_image_id = 'f851f66d-5d51-412c-a8a6-2e00c51bfb17'::uuid))
                                       Buffers: shared hit=3
               ->  Index Scan using container_remote_manifests_p37_pkey on container_remote_manifests_p37 container_remote_manifests_1  (cost=0.28..7.90 rows=1 width=26) (actual time=0.004..0.004 rows=1 loops=10)
                     Index Cond: ((id = "ANY_subquery"."reap_chunk.id") AND (namespace_id = '17195797-24c3-4531-979f-a3b04b62287f'::uuid))
                     Buffers: shared hit=30
 Planning:
   Buffers: shared hit=209
 Planning Time: 2.478 ms
 Execution Time: 96.834 ms

container_remote_manifests_p_namespace_id_container_remot_idx47 is the partition-local child of index_container_remote_manifests_on_ns_id_image_id_last_dl_at. Three trigger lines are cut from this block. They repeat the first block's set at 35.961ms, 54.197ms and 6.076ms for 10 calls each.

Extra pass, 5000 rows all under one image (the non-selective shape):

   InitPlan 1
     ->  Index Only Scan using container_remote_images_p55_pkey on container_remote_images_p55 container_remote_images  (cost=0.28..8.30 rows=1 width=0) (actual time=0.046..0.047 rows=1 loops=1)
           Index Cond: ((id = '578569cc-98a0-4169-bc3e-cee7e99692d1'::uuid) AND (namespace_id = '6a6502c6-3fd3-45c6-adf3-20db800d425d'::uuid))
           Heap Fetches: 1
...
                           ->  Limit  (cost=0.00..0.42 rows=10 width=16) (actual time=0.010..0.012 rows=10 loops=1)
                                 Buffers: shared hit=2
                                 ->  Seq Scan on container_remote_manifests_p55 reap_chunk  (cost=0.00..211.00 rows=5000 width=16) (actual time=0.008..0.010 rows=10 loops=1)
                                       Filter: ((namespace_id = '6a6502c6-3fd3-45c6-adf3-20db800d425d'::uuid) AND (container_remote_image_id = '578569cc-98a0-4169-bc3e-cee7e99692d1'::uuid))
                                       Buffers: shared hit=2
 Planning Time: 3.137 ms
 Execution Time: 122.641 ms

This excerpt shows the access path only. Every row of the partition matches the image predicate here, so the LIMIT stops the Seq Scan at once and the full-scan estimate of 211.00 never runs.

datastore.ContainerRemoteManifestReaper.Reap

Summary: The plan matches the method's intent. All three conjuncts reach the row, namespace_id prunes to container_remote_manifests_p47, and the estimate matches reality at 1 / 1 row. At 5000 rows in the partition the planner takes index_container_remote_manifests_on_ns_id_soft_deleted_at_id, the partial tombstone index, with namespace_id, soft_deleted_at IS NOT NULL and id all as Index Cond. The statement carries no anomaly.

Seed shape: namespaces=1, repositories=1, container_remote_repositories=1, container_remote_images=50, blob_storage_blobs=51, blob_storage_attachments=51, container_remote_manifests=50

One of the 50 manifest rows carries a soft_deleted_at value. It is the target, and the other 49 are the rows the predicate has to reject.

Rendered SQL:

DELETE FROM public.container_remote_manifests
WHERE ((container_remote_manifests.namespace_id = $1::uuid) AND (container_remote_manifests.id = $2::uuid)) AND (container_remote_manifests.soft_deleted_at IS NOT NULL)
RETURNING container_remote_manifests.blob_storage_attachment_id AS "reaped_attachment.id",
          container_remote_manifests.blob_sha256 AS "reaped_attachment.sha256";

Bound args: [82ea38f9-0855-4fac-9918-54c55ba6afab, 58af9bc0-df5d-43f1-be70-acd9a00acb1e]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Delete on container_remote_manifests  (cost=0.00..3.75 rows=1 width=10) (actual time=0.142..0.146 rows=1 loops=1)
   Delete on container_remote_manifests_p47 container_remote_manifests_1
   Buffers: shared hit=24
   ->  Seq Scan on container_remote_manifests_p47 container_remote_manifests_1  (cost=0.00..3.75 rows=1 width=10) (actual time=0.011..0.015 rows=1 loops=1)
         Filter: ((soft_deleted_at IS NOT NULL) AND (namespace_id = '82ea38f9-0855-4fac-9918-54c55ba6afab'::uuid) AND (id = '58af9bc0-df5d-43f1-be70-acd9a00acb1e'::uuid))
         Rows Removed by Filter: 49
         Buffers: shared hit=3
 Planning:
   Buffers: shared hit=48
 Planning Time: 0.489 ms
 Trigger for constraint container_remote_tags_container_remote_manifest_id_names_fkey47 on container_remote_manifests_p47: time=6.823 calls=1
 Trigger for constraint container_remote_manifest_re_parent_container_remote_man_fkey47 on container_remote_manifests_p47: time=8.163 calls=1
 Trigger for constraint container_remote_manifest_re_child_container_remote_mani_fkey47 on container_remote_manifests_p47: time=0.576 calls=1
 Execution Time: 15.964 ms

Timings: planning 0.489ms, execution 15.964ms, total 16.453ms.

Extra pass, 5000 rows in the one partition:

 Delete on container_remote_manifests  (cost=0.12..8.15 rows=1 width=10) (actual time=0.213..0.215 rows=1 loops=1)
   Delete on container_remote_manifests_p49 container_remote_manifests_1
   Buffers: shared hit=23
   ->  Index Scan using container_remote_manifests_p4_namespace_id_soft_deleted_at_idx9 on container_remote_manifests_p49 container_remote_manifests_1  (cost=0.12..8.15 rows=1 width=10) (actual time=0.026..0.027 rows=1 loops=1)
         Index Cond: ((namespace_id = 'cdf3b8ea-7430-47da-bfbd-228cb8d74b71'::uuid) AND (soft_deleted_at IS NOT NULL) AND (id = 'fb3d0360-c6f4-4030-8af2-aaaeada95317'::uuid))
         Buffers: shared hit=2
 Planning:
   Buffers: shared hit=108
 Planning Time: 2.159 ms
 Trigger for constraint container_remote_tags_container_remote_manifest_id_names_fkey49 on container_remote_manifests_p49: time=2.017 calls=1
 Trigger for constraint container_remote_manifest_re_parent_container_remote_man_fkey49 on container_remote_manifests_p49: time=1.860 calls=1
 Trigger for constraint container_remote_manifest_re_child_container_remote_mani_fkey49 on container_remote_manifests_p49: time=0.176 calls=1
 Execution Time: 4.653 ms

container_remote_manifests_p4_namespace_id_soft_deleted_at_idx9 is the partition-local child of index_container_remote_manifests_on_ns_id_soft_deleted_at_id.

datastore.deleteTombstonedContainerRemoteImage

Summary: The plan matches the method's intent. namespace_id prunes to container_remote_images_p56 and the id reaches the row. At 5000 image rows in the partition the planner takes pk_container_remote_images with both columns as Index Cond. The rows=0 figure on the Delete node is the shape of a DELETE with no RETURNING and not a miss. The statement carries no anomaly.

Seed shape: namespaces=1, repositories=1, container_remote_repositories=1, container_remote_images=50, blob_storage_blobs=51, blob_storage_attachments=51, container_remote_manifests=50

The bound image is image 2, which holds no manifest row. That is the state the reap reaches this statement in, because fk_container_remote_manifests_container_remote_image_id declares no ON DELETE action and a surviving child aborts the chunk with SQLSTATE 23503.

Rendered SQL:

DELETE FROM public.container_remote_images
WHERE (container_remote_images.namespace_id = $1::uuid) AND (container_remote_images.id = $2::uuid);

Bound args: [309cdee9-d961-49f3-8af4-fab44ea01132, d6b0b6b6-b88c-49bf-b2b3-7c879ce7207e]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Delete on container_remote_images  (cost=0.00..1.75 rows=0 width=0) (actual time=0.331..0.332 rows=0 loops=1)
   Delete on container_remote_images_p56 container_remote_images_1
   Buffers: shared hit=24
   ->  Seq Scan on container_remote_images_p56 container_remote_images_1  (cost=0.00..1.75 rows=1 width=10) (actual time=0.016..0.023 rows=1 loops=1)
         Filter: ((namespace_id = '309cdee9-d961-49f3-8af4-fab44ea01132'::uuid) AND (id = 'd6b0b6b6-b88c-49bf-b2b3-7c879ce7207e'::uuid))
         Rows Removed by Filter: 49
         Buffers: shared hit=1
 Planning:
   Buffers: shared hit=53
 Planning Time: 1.158 ms
 Trigger for constraint container_remote_manifests_container_remote_image_id_nam_fkey56 on container_remote_images_p56: time=2.067 calls=1
 Trigger for constraint container_remote_blobs_container_remote_image_id_namespa_fkey56 on container_remote_images_p56: time=7.352 calls=1
 Trigger for constraint container_remote_tags_container_remote_image_id_namespac_fkey56 on container_remote_images_p56: time=3.674 calls=1
 Trigger for constraint container_remote_manifest_re_container_remote_image_id_n_fkey56 on container_remote_images_p56: time=3.618 calls=1
 Execution Time: 17.634 ms

Timings: planning 1.158ms, execution 17.634ms, total 18.792ms.

Extra pass, 5000 image rows in the one partition:

 Delete on container_remote_images  (cost=0.28..8.30 rows=0 width=0) (actual time=0.121..0.122 rows=0 loops=1)
   Delete on container_remote_images_p16 container_remote_images_1
   Buffers: shared hit=26
   ->  Index Scan using container_remote_images_p16_pkey on container_remote_images_p16 container_remote_images_1  (cost=0.28..8.30 rows=1 width=10) (actual time=0.012..0.013 rows=1 loops=1)
         Index Cond: ((id = '522a345f-2ace-4a26-8bed-4f5fc8b88da2'::uuid) AND (namespace_id = '3f61a3b7-7ade-418b-a8c9-3a35e656e795'::uuid))
         Buffers: shared hit=3
 Planning:
   Buffers: shared hit=53
 Planning Time: 0.396 ms
 Trigger for constraint container_remote_manifests_container_remote_image_id_nam_fkey16 on container_remote_images_p16: time=0.658 calls=1
 Trigger for constraint container_remote_blobs_container_remote_image_id_namespa_fkey16 on container_remote_images_p16: time=3.410 calls=1
 Trigger for constraint container_remote_tags_container_remote_image_id_namespac_fkey16 on container_remote_images_p16: time=2.169 calls=1
 Trigger for constraint container_remote_manifest_re_container_remote_image_id_n_fkey16 on container_remote_images_p16: time=1.543 calls=1
 Execution Time: 8.089 ms

How the evidence was collected

Diff base is git merge-base origin/main HEAD, which is 3f0d92e652583d54b48efa2449d4a2b5ccd41f18 at the time of writing. The base is named by command rather than by SHA, because every rebase moves it: earlier versions of this line named 3ddc1505 and then d5d9bcec, and both went stale. The evidence below was collected against a base that predates the latest rebases; nothing in internal/datastore/migrations/ has changed in any of them, so the migration claim holds at either base.
Changed query files by content are internal/datastore/lifecycle_reap_container_remote.go and internal/datastore/blob_storage_attachments.go. internal/datastore/query_names.go changed too and holds no dispatch call, so it is not a query file.
The three jet statements were rendered through a generated .Sql() harness in package datastore. The harness ran under -tags=development_stubs rather than the repository default -tags=integration,development_stubs, because the package's integration TestMain needs a live database and the harness needs none. The harness is deleted at the end of the run.
deleteAttachmentUnreferencedByContainerRemoteStmt is a raw const, so its SQL is quoted from the source. Nothing renders it.
Seed data, the ephemeral container and the harness are all removed. The container ran under a name derived from this worktree's path.
datastore.deleteTombstonedContainerRemoteImage, named in the tables and plans above, is datastore.deleteContainerRemoteImage on the branch now. The symbol was renamed after this evidence was collected, because the statement it builds tests no soft_deleted_at. The statement itself is unchanged, so every plan above still describes what runs.

Related to #611

Closes #741 (closed)

This is a bot message 🤖 — /smurfit

Edited Aug 20, 2026 by Pawel Rozlach

chore(datastore): container remote reap (S20-A plan: 12/21)

What this step delivers

The image reap is state-blind

ReapTotals.SizeBytes is zero in both reapers

Four rows corrected in the spec-coverage table

Spec coverage

Acceptance criteria

Error cases

Security considerations

e2e scenario catalogs

Diff size

Merge-order notes

1. The const-block realignment: steps 8 and 9 merged, this MR rebased onto both

2. The query-name budget

3. What S20-A step 9 settled for this MR

4. Open siblings, and the two conflicts a resolver will meet

Notes for the reviewer

Four sibling spec claims are left standing, deliberately

No e2e scenario is added or affected

The plan Status row

Eight files sit outside the plan's Files list

One spec criterion is deferred

The short-chunk comment is corrected, with no behaviour change

The reap's four statements carry a partition-pruning suite

One comment diverges from the plan's wording, deliberately

ADR freshness

Database Review Evidence

Queries

How the evidence was collected

Merge request reports

`ReapTotals.SizeBytes` is zero in both reapers

Eight files sit outside the plan's `Files` list