chore(managementapi): apply container bulk manifest batches in dependency order

Hayley Swimelarrequested to merge
313/step-31-container-bulk-worker-manifests into main

Why

The container bulk worker applies images and tags batches but answers the manifests collection with a typed cancel, so the last container bulk route (the manifests bulk_delete of the write-routes step) has no worker arm to enqueue against. Manifests are the one collection whose entries reference each other: an index blocks its children's deletion, so a per-entry pass in payload or list order strands exactly the rows it was asked to remove. This MR adds the manifests pass with the amended bulk-manifests semantics: dependency-ordered application (indexes before the manifests they index), blocked entries skipped and logged with their blocking parents, referrers included in delete_all.

What (non-obvious parts)

  • Dependency order is discovered, not precomputed. The subset pass re-attempts entries the cascade refused for as long as any entry applies, because deleting an index removes the relationship rows blocking its children. A batch naming an index and its child deletes both whichever order the payload lists them, nested chains drain to a fixpoint, and no relationship-graph query is needed. An entry still blocked when a sweep applies nothing is referenced from outside the batch: skipped, logged with its parents, left listable for the caller's re-read.
  • delete_all pages two classes. datastore.ContainerBulkScope gains two keyset reads splitting the image's manifests by whether they index others (EXISTS on the relationships parent side). The drain empties the index class first, so the common shape completes in one round, and runs further rounds only while progress continues (an index can itself be blocked by a super-index, or by one stamped after acceptance). Memory stays at one page: blocked rows remain in the collection and are re-paged, never retained.
  • Plan Files deviation. The plan lists only bulk_container_worker.go (Modify), but a delete_all pass has to enumerate the collection and SQL lives in the datastore per ADR-023, so the scope composer and the wire seam gain the manifests reads alongside.
  • No new blocking logic. The check is DeleteManifestByDigest's own referential-integrity read, unchanged; one production deleter now backs the tags and manifests seams.

Diff size: ~1,700 added lines, ~70% tests (unit fakes + suite ~860, integration ~350, wire seam test ~75; production ~560 across the worker pass, the two composer reads, and wiring). Splitting the pass from its scope reads would ship a pass that cannot drain, so the step ships whole per the plan.

Test plan

  • Unit: both payload orders (child-first kills a payload-order pass), nested index fixpoint, outside-index skip with logged parents, all-blocked batch reports success and deletes nothing, erroring index leaves its child skipped and aggregates once, index-class-first single-round drain, multi-page digest keyset pinned via recorded cursors, out-of-scope index blocks its child without failing the job, image-gone no-op for the manifests collection, one event per applied entry and none for skips.
  • Integration (real Postgres): both payload orders through the production cascade, blocked manifest survives and stays listable through the real list handler, delete_all drains index+child+referrer+boundary and leaves the post-acceptance survivor, composer-level page walk pinning class split, referrer inclusion, acceptance bound, and digest keyset.
  • Gates: go build ./..., unit and integration suites green, golangci-lint run --build-tags=integration --max-same-issues=0 --max-issues-per-linter=0 clean on touched files.

No e2e catalog edit: the delete-family scenario rows ride E2e lifecycle catalogs: one delete-family pass ... (#638) • Unassigned, with no per-arm edits.

Stacked on chore(managementapi): apply container bulk dele... (!1555 - merged) • Hayley Swimelar • 19.3.

Related to #313 (closed)

Database Review Evidence

Query-mode evidence for this MR's stacked diff against 313/step-30-container-bulk-worker: the two container manifest digest-scope reads. The diff introduces no migration; the container images bulk-scope index migration ships with chore(managementapi): apply container bulk dele... (!1555 - merged) • Hayley Swimelar • 19.3.

Queries

Note

Plans are from EXPLAIN (ANALYZE, BUFFERS) against an ephemeral PostgreSQL 17 container (matching GL_PG_CURR_VERSION from .gitlab-ci-other-versions.yml), with synthesized seed data rolled back per query and the container torn down at the end of the run. Numbers reflect moderate cardinality and do not capture production-scale effects. See Database review evidence for seed sizing, methodology, and the anomalies the skill flags. Expand each row's details for the seed shape, rendered SQL, bound args, and raw plan.

Method Plan node Index Rows (plan / actual) Cost Time Buffers (hit / read) Partitions
datastore.IndexManifestDigestsInScope.AfterDigestEmpty Limit pk_container_manifests 100 / 100 391.84 1.420ms 1560 / 0 1/64, 1/64
datastore.IndexManifestDigestsInScope.AfterDigestNonEmpty Limit pk_container_manifests 100 / 100 388.02 1.472ms 1560 / 0 1/64, 1/64
datastore.NonIndexManifestDigestsInScope.AfterDigestEmpty Limit unique_container_manifests_ns_id_ci_id_digest 100 / 100 82.67 0.167ms 239 / 0 1/64, 1/64
datastore.NonIndexManifestDigestsInScope.AfterDigestNonEmpty Limit unique_container_manifests_ns_id_ci_id_digest 100 / 100 105.24 0.168ms 240 / 0 1/64, 1/64

The two statements dispatch under their own query names, container_manifests_select_index_digests_in_scope and container_manifests_select_non_index_digests_in_scope. The Partitions cell pairs are container_manifests, container_manifest_relationships; every plan pruned both tables to one of 64 hash partitions at plan time. Index names in the plans are the per-partition children (63-char truncated) of the parent indexes named here.

datastore.IndexManifestDigestsInScope.AfterDigestEmpty

Summary: Plan matches the first-page intent: both tables prune to one of 64 partitions and the statement returns the first 100 index-manifest digests. At this cardinality the planner inverts the EXISTS instead of walking the digest index: it collects the namespace's 500 distinct parents from the relationships partition (Seq Scan + HashAggregate, 60 buffers), joins each back through pk_container_manifests, and top-N sorts by digest, so the sort input is bounded by the parent set, not the manifest count. The digest-ordered index the keyset rides exists and drives the NOT EXISTS arm's plans below. Estimates are exact (500 planned / 500 actual at the join), zero disk reads. No anomalies.

Seed shape: namespaces=1, repositories=1, container_repositories=1, container_images=1, blob_storage_blobs=1, blob_storage_attachments=1, container_manifests=5000, container_manifest_relationships=4500

All 5000 manifests sit under one image in one namespace (one hash partition). Every 10th manifest by digest order is an index parenting the 9 manifests before it: 500 index parents, 4500 children.

Rendered SQL:

SELECT container_manifests.digest AS "scope_row.digest"
FROM public.container_manifests
WHERE (((container_manifests.namespace_id = $1::uuid) AND (container_manifests.container_image_id = $2::uuid)) AND (container_manifests.created_at <= $3::timestamp with time zone)) AND (EXISTS (
           SELECT $4
           FROM public.container_manifest_relationships
           WHERE (container_manifest_relationships.namespace_id = container_manifests.namespace_id) AND (container_manifest_relationships.parent_container_manifest_id = container_manifests.id)
      ))
ORDER BY container_manifests.digest ASC
LIMIT $5;

Bound args: [9208e831-4593-4260-81f9-9a027af208be, c6cb2193-34ac-451d-983e-68b9b2161049, 2026-08-18 00:13:46.478908+00, 1, 100] ($3 is the seeding transaction's now(), so every seeded row sits exactly on the inclusive <= boundary; $5 is containerBulkPageSize = 100, the value both call sites pass)

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Limit  (cost=391.59..391.84 rows=100 width=33) (actual time=1.413..1.420 rows=100 loops=1)
   Buffers: shared hit=1560
   ->  Sort  (cost=391.59..392.84 rows=500 width=33) (actual time=1.412..1.415 rows=100 loops=1)
         Sort Key: container_manifests.digest
         Sort Method: top-N heapsort  Memory: 2kB
         Buffers: shared hit=1560
         ->  Nested Loop  (cost=139.03..372.48 rows=500 width=33) (actual time=0.739..1.308 rows=500 loops=1)
               Buffers: shared hit=1560
               ->  HashAggregate  (cost=138.75..143.75 rows=500 width=32) (actual time=0.728..0.767 rows=500 loops=1)
                     Group Key: container_manifest_relationships.namespace_id, container_manifest_relationships.parent_container_manifest_id
                     Batches: 1  Memory Usage: 105kB
                     Buffers: shared hit=60
                     ->  Seq Scan on container_manifest_relationships_p15 container_manifest_relationships  (cost=0.00..116.25 rows=4500 width=32) (actual time=0.005..0.293 rows=4500 loops=1)
                           Filter: (namespace_id = '9208e831-4593-4260-81f9-9a027af208be'::uuid)
                           Buffers: shared hit=60
               ->  Index Scan using container_manifests_p15_pkey on container_manifests_p15 container_manifests  (cost=0.28..0.46 rows=1 width=65) (actual time=0.001..0.001 rows=1 loops=500)
                     Index Cond: ((id = container_manifest_relationships.parent_container_manifest_id) AND (namespace_id = '9208e831-4593-4260-81f9-9a027af208be'::uuid))
                     Filter: ((created_at <= '2026-08-18 00:13:46.478908+00'::timestamp with time zone) AND (container_image_id = 'c6cb2193-34ac-451d-983e-68b9b2161049'::uuid))
                     Buffers: shared hit=1500
 Planning:
   Buffers: shared hit=599
 Planning Time: 1.898 ms
 Execution Time: 1.459 ms

Timings: planning 1.898ms, execution 1.459ms, total 3.357ms.

datastore.IndexManifestDigestsInScope.AfterDigestNonEmpty

Summary: Same inverted-EXISTS shape as the first page, with the digest cursor (row 2500 of 5000) applied as a filter on the join probe: 250 parents survive digest > $5 and the top-N sort emits 100, in 1.5ms with zero disk reads. The cursor lands as a filter rather than an index bound in this shape because the relationships side drives the join; the anti-join arm below shows the cursor as an index bound. No anomalies.

Seed shape: namespaces=1, repositories=1, container_repositories=1, container_images=1, blob_storage_blobs=1, blob_storage_attachments=1, container_manifests=5000, container_manifest_relationships=4500

Same layout as the first-page read: 500 index parents (every 10th digest), 4500 children.

Rendered SQL:

SELECT container_manifests.digest AS "scope_row.digest"
FROM public.container_manifests
WHERE ((((container_manifests.namespace_id = $1::uuid) AND (container_manifests.container_image_id = $2::uuid)) AND (container_manifests.created_at <= $3::timestamp with time zone)) AND (EXISTS (
           SELECT $4
           FROM public.container_manifest_relationships
           WHERE (container_manifest_relationships.namespace_id = container_manifests.namespace_id) AND (container_manifest_relationships.parent_container_manifest_id = container_manifests.id)
      ))) AND (container_manifests.digest > $5::bytea)
ORDER BY container_manifests.digest ASC
LIMIT $6;

Bound args: [9cf98837-3b37-47cc-b0d1-c923b1ba3cfa, 2d6fb73d-8782-4598-a7ac-14a5f2cdac44, 2026-08-18 00:14:59.349646+00, 1, \x00000000000000000000000000000000000000000000000000000000000009c4, 100] ($5 is the mid-range digest cursor, row 2500 of the 5000 seeded digests)

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Limit  (cost=387.77..388.02 rows=100 width=33) (actual time=1.465..1.472 rows=100 loops=1)
   Buffers: shared hit=1560
   ->  Sort  (cost=387.77..389.02 rows=500 width=33) (actual time=1.464..1.467 rows=100 loops=1)
         Sort Key: container_manifests.digest
         Sort Method: top-N heapsort  Memory: 19kB
         Buffers: shared hit=1560
         ->  Nested Loop  (cost=139.03..368.66 rows=500 width=33) (actual time=0.762..1.387 rows=250 loops=1)
               Buffers: shared hit=1560
               ->  HashAggregate  (cost=138.75..143.75 rows=500 width=32) (actual time=0.742..0.782 rows=500 loops=1)
                     Group Key: container_manifest_relationships.namespace_id, container_manifest_relationships.parent_container_manifest_id
                     Batches: 1  Memory Usage: 105kB
                     Buffers: shared hit=60
                     ->  Seq Scan on container_manifest_relationships_p48 container_manifest_relationships  (cost=0.00..116.25 rows=4500 width=32) (actual time=0.005..0.310 rows=4500 loops=1)
                           Filter: (namespace_id = '9cf98837-3b37-47cc-b0d1-c923b1ba3cfa'::uuid)
                           Buffers: shared hit=60
               ->  Index Scan using container_manifests_p48_pkey on container_manifests_p48 container_manifests  (cost=0.28..0.47 rows=1 width=65) (actual time=0.001..0.001 rows=0 loops=500)
                     Index Cond: ((id = container_manifest_relationships.parent_container_manifest_id) AND (namespace_id = '9cf98837-3b37-47cc-b0d1-c923b1ba3cfa'::uuid))
                     Filter: ((created_at <= '2026-08-18 00:14:59.349646+00'::timestamp with time zone) AND (digest > '\x00000000000000000000000000000000000000000000000000000000000009c4'::bytea) AND (container_image_id = '2d6fb73d-8782-4598-a7ac-14a5f2cdac44'::uuid))
                     Rows Removed by Filter: 0
                     Buffers: shared hit=1500
 Planning:
   Buffers: shared hit=198
 Planning Time: 0.971 ms
 Execution Time: 1.522 ms

Timings: planning 0.971ms, execution 1.522ms, total 2.493ms.

datastore.NonIndexManifestDigestsInScope.AfterDigestEmpty

Summary: Exactly the keyset shape the method's doc comment claims: a digest-ordered Index Scan over the partition child of unique_container_manifests_ns_id_ci_id_digest with equality on namespace_id and container_image_id, anti-joined per row against the relationships unique triple's (namespace_id, parent_container_manifest_id) prefix as an Index Only Scan, stopping at the LIMIT after walking 111 index entries to emit 100 (the 11 skipped rows are the index parents in that range). 0.19ms execution, 239 buffers, zero disk reads. No anomalies.

Seed shape: namespaces=1, repositories=1, container_repositories=1, container_images=1, blob_storage_blobs=1, blob_storage_attachments=1, container_manifests=5000, container_manifest_relationships=4500

Same layout as the EXISTS arm: 500 index parents (every 10th digest), 4500 children.

Rendered SQL:

SELECT container_manifests.digest AS "scope_row.digest"
FROM public.container_manifests
WHERE (((container_manifests.namespace_id = $1::uuid) AND (container_manifests.container_image_id = $2::uuid)) AND (container_manifests.created_at <= $3::timestamp with time zone)) AND (NOT (EXISTS (
           SELECT $4
           FROM public.container_manifest_relationships
           WHERE (container_manifest_relationships.namespace_id = container_manifests.namespace_id) AND (container_manifest_relationships.parent_container_manifest_id = container_manifests.id)
      )))
ORDER BY container_manifests.digest ASC
LIMIT $5;

Bound args: [6a0cbea0-3e78-4d16-ab24-3657f5d2fa52, 8c07ba32-f797-4e59-8194-3b109a105f71, 2026-08-18 00:15:19.273781+00, 1, 100]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Limit  (cost=0.56..82.67 rows=100 width=33) (actual time=0.019..0.167 rows=100 loops=1)
   Buffers: shared hit=239
   ->  Nested Loop Anti Join  (cost=0.56..3695.47 rows=4500 width=33) (actual time=0.018..0.162 rows=100 loops=1)
         Buffers: shared hit=239
         ->  Index Scan using container_manifests_p63_namespace_id_container_image_id_dig_idx on container_manifests_p63 container_manifests  (cost=0.28..672.97 rows=5000 width=65) (actual time=0.011..0.029 rows=111 loops=1)
               Index Cond: ((namespace_id = '6a0cbea0-3e78-4d16-ab24-3657f5d2fa52'::uuid) AND (container_image_id = '8c07ba32-f797-4e59-8194-3b109a105f71'::uuid))
               Filter: (created_at <= '2026-08-18 00:15:19.273781+00'::timestamp with time zone)
               Buffers: shared hit=6
         ->  Index Only Scan using container_manifest_relations_namespace_id_parent_containe_idx63 on container_manifest_relationships_p63 container_manifest_relationships  (cost=0.28..0.54 rows=9 width=32) (actual time=0.001..0.001 rows=0 loops=111)
               Index Cond: ((namespace_id = '6a0cbea0-3e78-4d16-ab24-3657f5d2fa52'::uuid) AND (parent_container_manifest_id = container_manifests.id))
               Heap Fetches: 11
               Buffers: shared hit=233
 Planning:
   Buffers: shared hit=182
 Planning Time: 0.822 ms
 Execution Time: 0.187 ms

Timings: planning 0.822ms, execution 0.187ms, total 1.009ms.

datastore.NonIndexManifestDigestsInScope.AfterDigestNonEmpty

Summary: Same anti-join walk as the first page, with the digest cursor bound inside the Index Cond (digest > $5 rides the index range rather than a post-scan filter), which is the keyset resume behaving as designed. Walks 111 index entries for 100 rows out, 0.19ms, zero disk reads. No anomalies.

Seed shape: namespaces=1, repositories=1, container_repositories=1, container_images=1, blob_storage_blobs=1, blob_storage_attachments=1, container_manifests=5000, container_manifest_relationships=4500

Same layout as the EXISTS arm: 500 index parents (every 10th digest), 4500 children.

Rendered SQL:

SELECT container_manifests.digest AS "scope_row.digest"
FROM public.container_manifests
WHERE ((((container_manifests.namespace_id = $1::uuid) AND (container_manifests.container_image_id = $2::uuid)) AND (container_manifests.created_at <= $3::timestamp with time zone)) AND (NOT (EXISTS (
           SELECT $4
           FROM public.container_manifest_relationships
           WHERE (container_manifest_relationships.namespace_id = container_manifests.namespace_id) AND (container_manifest_relationships.parent_container_manifest_id = container_manifests.id)
      )))) AND (container_manifests.digest > $5::bytea)
ORDER BY container_manifests.digest ASC
LIMIT $6;

Bound args: [5e9d148b-a1e7-4d46-8754-9c57618c9efa, 32e4794b-dfc7-41d6-b658-b0b7c65ef92f, 2026-08-18 00:15:39.421472+00, 1, \x00000000000000000000000000000000000000000000000000000000000009c4, 100] ($5 is the mid-range digest cursor, row 2500 of the 5000 seeded digests)

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

 Limit  (cost=0.56..105.24 rows=100 width=33) (actual time=0.019..0.168 rows=100 loops=1)
   Buffers: shared hit=240
   ->  Nested Loop Anti Join  (cost=0.56..2094.13 rows=2000 width=33) (actual time=0.018..0.163 rows=100 loops=1)
         Buffers: shared hit=240
         ->  Index Scan using container_manifests_p43_namespace_id_container_image_id_dig_idx on container_manifests_p43 container_manifests  (cost=0.28..464.72 rows=2500 width=65) (actual time=0.011..0.029 rows=111 loops=1)
               Index Cond: ((namespace_id = '5e9d148b-a1e7-4d46-8754-9c57618c9efa'::uuid) AND (container_image_id = '32e4794b-dfc7-41d6-b658-b0b7c65ef92f'::uuid) AND (digest > '\x00000000000000000000000000000000000000000000000000000000000009c4'::bytea))
               Filter: (created_at <= '2026-08-18 00:15:39.421472+00'::timestamp with time zone)
               Buffers: shared hit=7
         ->  Index Only Scan using container_manifest_relations_namespace_id_parent_containe_idx43 on container_manifest_relationships_p43 container_manifest_relationships  (cost=0.28..0.63 rows=9 width=32) (actual time=0.001..0.001 rows=0 loops=111)
               Index Cond: ((namespace_id = '5e9d148b-a1e7-4d46-8754-9c57618c9efa'::uuid) AND (parent_container_manifest_id = container_manifests.id))
               Heap Fetches: 11
               Buffers: shared hit=233
 Planning:
   Buffers: shared hit=185
 Planning Time: 0.852 ms
 Execution Time: 0.190 ms

Timings: planning 0.852ms, execution 0.190ms, total 1.042ms.

Edited by Hayley Swimelar

Merge request reports