feat(oci): Step 11 part 1 - record download signals for cached container remote reads

Why

S16's remote read arms have to record that a pull happened: last_downloaded_at is what cache eviction reads, so a remote cache that never writes it runs a retention sweep that reclaims its hottest layers first. This MR ships the seam those arms will record through, and the configuration field its throttle reads.

Step 11a of the S16 container remote plan, specified by S16.

Step 11 ships as three sequential MRs, split for review size — the whole step forecasts several thousand lines against the development model's 500-line ceiling. This is 11a: the download signals. 11b adds the shared response-header writer, the digest gate, and the manifest arm; 11c adds the blob arm. Each branches off main after the previous one merges, so no branch is stacked and each diff against main equals its own change.

What (the non-obvious parts)

• The seam ships with no production caller, and that is the point. BumpLastDownloadedAt is one of the four methods remote.CacheStore requires, and Step 13 constructs Lookuper and Fetcher over that full interface — so until the datastore method exists, Step 13 does not compile. Putting it in the first of the three MRs gets it onto main at the earliest point the split allows. Step 8 added BumpUpstreamCheckedAt with no caller of its own for the same reason. The method alone does not finish the interface: the wrapper edit that presents it to remote.CacheStore is 11b's, because internal/format/oci/remote_cache_store.go does not exist on main until Step 8's oci MR lands, so this MR cannot edit it. Step 13 therefore depends on 11b rather than on this MR, and no wire_oci.go is touched here.

• last_downloaded_at_staleness_window is validated at startup and not otherwise read yet. This MR ships the field end to end at the config boundary — proto field 5, regenerated gen/**, the 1h loader default, both startup checks, config.example.yaml, and the configuration reference — but the store that reads the throttle holds its own copy of the 1h default, because this MR constructs no store and so has no site to hand the operator's value to. A configured 6h passes every check and leaves the throttle at 1h. 11b closes it, since it adds the first NewContainerRemoteCacheStore call on a production wiring path. Both doc comments name the other copy and the configuration reference says the field is validated and not otherwise read, so the gap is stated where a reader will look rather than discovered. No test can catch this either way, which is why it is called out rather than asserted: a test can only assert the window the store was given.

• The 24h ceiling is derived, not chosen. ADR-010 puts the cache retention period at a one-day floor, and the throttle holds a manifest's timestamp still for up to one window — so at exactly 24h a continuously-pulled manifest's stored value reaches the retention cutoff. The check is therefore exclusive (>= 24h fails), evaluated at config-load time against a bare constant, reading no repository's own retention period.

• A manifest read is two statements, in an order chosen against the cache fill. The throttled container_remote_manifests bump runs first and returns its parent image id; the container_remote_images bump runs second, only when the first reported a row. Both tables are therefore gated on the manifest row's freshness rather than each on its own column — gating the image on its own would keep bumping it on reads whose manifest bump was skipped, which is the write the throttle exists to avoid.

  This was one data-modifying CTE, and the order is why it is not. Postgres runs such a CTE to completion before the enclosing UPDATE can produce a row to lock, so it took the manifest row's lock and then asked for the image's. upsertCacheFill goes the other way — upsertContainerRemoteImage's ON CONFLICT DO UPDATE holds the image lock to end of transaction and the manifest upsert comes later — so a bump racing a re-fill of the same digest deadlocks. Two statements on the pool are each their own transaction, so neither side holds one lock while waiting on the other. MavenRemoteCacheStore.BumpLastDownloadedAtByID and the npm remote bumps are both separate statements too; the CTE was the outlier.

  The trade is cross-table atomicity, and it is not free. A manifest bump whose image bump then fails leaves the manifest advanced and the image behind it. Maven carries the identical window between its parent and child bumps and documents it as accepted loss, on the same grounds: the columns are lossy retention signals and the next read rewrites both. The pair was never atomic under concurrency anyway — two reads racing the throttle can both find the row stale and both write, which GREATEST(existing, NOW()) is what makes harmless.

  Each statement carries namespace_id on the UPDATE itself, which is what prunes to one hash partition; the second one's predicate is a bare id equality, so without the literal it would Append over all 64 partitions of container_remote_images on every stale manifest read.

• The blob route's bump is unthrottled, per layer, against one row. A blob read has no manifest row to check freshness against, so extending the skip means keying it on the image row's own column. Not done here, and the cost is real: last_downloaded_at is a key column of the retention index, so no bump is heap-only, and a multi-layer pull rewrites the one image row once per layer. Named on the method and recorded in the spec's Follow-ups; tracked as surface 1 of #680.

• This diverges from ADR-007's own text, which nothing else recorded. ADR-007 flags container_remote_images.last_downloaded_at as buffered and says it is "maintained via buffered/async writes ... to avoid hot-row contention", defining that mechanism as an intermediate store merged back by a background process. No format has one — Maven, npm, and container remote all write the column directly on each served read. The code matches the shipped siblings, so the ADR's prose is what should move; this MR routes that into the standing handbook amendment #30 alongside the schema items, including the instruction not to publish the new manifests column as buffered.

• The downloads_count leg is not here; it lands with S18's writer. An earlier revision of this branch shipped it as a non-executing stub — a store method returning nil, a statement builder only a test called, a DownloadCounter interface, a second emission in the dispatcher, and a metering flag type whose only job was keeping that emission's result=ok off the metric. Review called it scaffolding over zero behavior, and the "never carries result=ok" exception it forced had reached four separate surfaces to explain a write that never ran. All of it is removed. The compile-order argument that keeps BumpLastDownloadedAt here does not extend to the counter: that method's SQL executes and is asserted against a live database, and the counter's never did.

  One consequence to carry into review: docs/specs/S16-container-remote.md still lists a downloads_count event on every cached read, gated on S18, and says this step wires each read to a store seam carrying the increment. That sentence no longer describes the code. The plan records the divergence; a spec amendment rather than this MR is what settles it.

• The dispatch is detached and bounded, following npm rather than Maven. The request context is canceled the moment the handler returns, so a context-aware Exec on it would report cancellation before touching the row and silently drop the signal for a serve that did complete. The write carries its own deadline and recovers its own panic — the goroutine outlives the handler, with no HTTP panic-recovery middleware on its stack. Dispatches are shed rather than queued past either of two caps, and result=dropped is what makes an undersized one observable, since the read still succeeds and nothing reaches an error log.

  The second cap is a per-repository sub-quota, 16 of the 64 process-wide slots. Without it the global cap is a shared resource one repository can take entirely, and the unthrottled blob route is what makes that reachable: every layer of an image resolves to the same container_remote_images row, so concurrent layer reads serialize on that row's lock while each holds a slot. Because the cap is process-wide and sheds rather than queues, the dispatches lost would be every other repository's, in every other namespace. Both siblings that meet this already carry a sub-quota — maven.bufferedUpdateMaxPerLabel across three fixed labels, npm.rebuildMaxPerPackage across an unbounded package space — and this follows npm's, since repositories are unbounded the same way. 16 rather than npm's 4 because one client's layer fetches are concurrent, so a smaller quota would shed an ordinary multi-layer pull; both numbers are heuristics that bound blast radius rather than model capacity.

  This is the fourth copy of the shed-pattern cap, after maven.bufferedUpdateMaxInFlight and npm's two in packument_cache.go and inline_build.go. ADR-023 forbids the cross-format import and explicitly defers shared scheduling infrastructure as premature, so copying is the accepted answer rather than an oversight — but copy four is what makes the eventual pool-aware fix a four-site change instead of three. Tracked in #559.

• The outcome metric sits outside this spec's oci_remote_* namespace on purpose. It goes on gitlab_artifact_registry_oci_buffered_counter_updates_total{column, result}, the format's interim buffered-write counter. The counter is deleted with the helper it meters when the cross-format pipeline ships, so publishing it in S16's metric table would put a metric with a scheduled end date in a permanent surface. Both siblings had a remote namespace of their own and filed the same write on their format-wide counter anyway. The result values follow Maven's four; npm's has no error, and the one column value is one Maven already emits, so neither label's cardinality budget changes.

Reviewable size

1,764 reviewable Go against the plan's Est. ~1,400 per MR — 1.26x. Counted the house way: added .go lines with blanks and comment-only lines stripped, gen/** excluded. Measured against main after the rebase below.

Group	Files	Raw added	Reviewable
Production Go	8	1,148	385
Test Go	9	2,589	1,379 (78%)
Proto + YAML	3	23	6
Docs (plan, spec, dev)	4	158	146
gen/** (generated, excluded)	1	23	—

Tests at 78% sit inside the 72–85% band the four sibling building-block MRs run at, so "most of it is tests" is not the argument — the band is. Mid-band against those siblings too: S15 npm Step 4a 1,387, S14 maven Step 5 1,533, S16 Step 3 1,555, this MR 1,764, S15 npm Step 4b 2,254.

Production Go is about 66% comment lines (1,148 raw to 385 counted). That is deliberate — most of it is rationale for decisions a reader would otherwise have to reconstruct (why the two bumps are separate statements and in which order, why the sub-quota is keyed on the repository, why the counter seam is not metered on success).

205 of those lines are a self-review round taken before this MR was assigned a reviewer, across nine commits. Two fixed defects rather than style: the manifest bump took its two row locks in the opposite order from upsertCacheFill, and a dispatcher test failed under -race three runs in five on the tree as first pushed. The rest close a fairness gap both siblings had already solved, stop a metric reporting a write that never runs, and correct four comments plus a wrong claim in the invariants scan's header. Splitting them out was considered and rejected: every one fixes code this MR itself introduces, so a split would mean a base MR knowingly carrying a deadlock, and the merged artifact is identical either way since this squashes.

Rebased after Step 8's datastore MR merged

This branch was rebased onto main after !1640 (merged) landed. Two files conflicted and both resolutions are worth a reviewer's eye:

• internal/datastore/query_names.go — a union. !1640 added the fill and attachment names and S17 added the eviction ones; this MR adds the two bump names. Both new names sort into the existing per-table groups rather than opening a second container_remote_images group.
• docs/plans/2026-07-30-container-remote.md — !1640 wrote Step 8's entry saying 11a carries the wrapper edit and the var _ remote.CacheStore anchor, and that Step 13 depends on 11a. That is the pre-split allocation; the current one puts the edit on 11b, because internal/format/oci/remote_cache_store.go does not exist on main until Step 8's oci MR (!1641 (merged), still open) lands, so 11a cannot edit it. This MR's resolution keeps the 11b allocation and folds in !1640's remoteCachePromoted / remoteCacheFiller naming. !1641 merging does not reopen the question — it is what makes 11b's edit authorable.

Still about 3x the 500-line threshold, so this section is owed. Why not split further: a split is possible — the datastore statements and the OCI dispatcher do not call each other in this MR — but it does not help. The step is already three MRs, and splitting 11a again would put the seam and its coverage in different reviews while leaving each half under-tested on its own.

Test plan

go test -count=1 ./internal/datastore/ ./internal/format/oci/ ./internal/config/ ./internal/metrics/
go test -count=1 -tags=integration ./internal/datastore/
golangci-lint run ./internal/format/oci/... ./internal/config/... ./internal/metrics/...
golangci-lint run --build-tags=integration --max-same-issues=0 --max-issues-per-linter=0 ./internal/datastore/...

The bump statements are asserted against a live database: a manifest bump writes both timestamps and a blob bump only the image timestamp, an earlier clock never moves a timestamp backwards, and a repeat manifest read inside the window issues no UPDATE on either row. That last one asserts the absence of the write through xmin row-version equality on both tables, not the stability of the value, which is what the criterion actually asks for. Both parent formats (docker and oci) are covered as positive hits rather than one standing in for the other.

Two predicates no live fixture can reach are pinned in rendered SQL instead. The throttle's <= versus < differ only when the timestamp equals NOW() minus the window exactly. And the manifest column's GREATEST guard is unreachable because a row holding a future timestamp is inside the window, so the throttle skips the statement before the SET is evaluated — a plain NOW() there passed the whole integration suite before this assertion existed.

A separate EXPLAIN suite proves every statement prunes to a single hash partition of each table it touches, running the store's own builders rather than a hand-maintained copy. The write side needs this more than the reads do: the namespace literal has to survive propagation into a correlated EXISTS and onto an UPDATE's own target table, and losing it at either point plans an Append over all 64 partitions while every behavioral subtest still passes. The manifest route's image bump is asserted separately from the others, because it reads container_remote_images alone — the shared parent chain is not in its plan — and because its predicate is a bare id equality that looks incapable of scanning anything until you remember the table is partitioned by namespace_id and not by id.

Two contracts are pinned that no other assertion reaches. A read on an already-canceled context reports context.Canceled and issues no UPDATE on either row, on both routes: the dispatcher strips cancellation with context.WithoutCancel precisely because a context-aware Exec would drop the signal for a read that did reach the client, so a store that swallowed the cancellation would make it report a delivered write that never ran. And a delivered read meters result="ok", which is what gives the dropped and error series a denominator and is asserted nowhere else in the suite.

The integration-tagged lint run reports 4,083 findings across internal/datastore; filtered to this MR's files it is one — a modernize newexpr on ptr(x), left as is because the mirrored sibling suite carries ten identical hits and new(expr) needs a newer Go than this module pins. Every //nolint token here was measured by removing it and re-running rather than inherited from a sibling — including the staticcheck token on the nil-context test, which fires SA1012, and the paralleltest tokens, which name the process-global bufferedCounterUpdates vector they are serial for. Two gochecknoglobals tokens that suppressed nothing, because the linter is disabled repo-wide, were removed on this review. There is no dupl token, because this file's want maps never reach its threshold.

No e2e scenario is added or affected: this MR adds no route, mounts no handler, and serves no request, so docs/testing/ has nothing to exercise until 11b and 11c wire the arms. Container/OCI conformance is likewise unreached for the same reason and is left to CI.

Review passes

/implement-step, /simplify, and two /validate-step passes. The second validation pass found four blockers, all in text rather than behavior, and all fixed here: the plan's Step 8 entry still described the pre-split allocation (that 11a carries the wrapper anchor and that Step 13 depends on 11a); the spec claimed neither sibling has a remote metric namespace when both declare one; a five-table write-amplification claim survived in the Download signals body after being corrected in two other places; and the ADR-007 mechanism divergence was named in the spec but not carried by the amendment item the spec points at.

It also found three coverage gaps, each covering a predicate that could be deleted with the suite still green: the manifest GREATEST guard, the new metric vector's absence from two tests whose docs claim to cover every vector, and the log-level split plus identifier assertions both siblings carry. All three are closed, and each new assertion was mutation-checked — the production edit that should break it does.

Database Review Evidence

Queries

Note

Plans are from EXPLAIN (ANALYZE, BUFFERS) against an ephemeral PostgreSQL 17 container (17.10, matching GL_PG_CURR_VERSION from .gitlab-ci-other-versions.yml), with synthesized seed data rolled back per query and the container torn down at the end of the run. Numbers reflect moderate cardinality and do not capture production-scale effects. See Database review evidence for seed sizing, methodology, and the anomalies the skill flags. Expand each row's details for the seed shape, rendered SQL, bound args, and raw plan.

No migration files changed on this branch, so migration mode did not run and there is no ### Migrations subsection. The three statement shapes under review all come from internal/datastore/container_remote_download.go; container_remote_cache.go and container_remote_cache_write.go also changed but only in comments, so no statement they build moved and neither contributes a row here.

Method	Plan node	Index	Rows (plan / actual)	Cost	Time	Buffers (hit / read)	Partitions
datastore.containerRemoteBumpManifestDownloadedAtStmt.TagKind	Update	container_remote_tags_p20_namespace_id_container_remote_ima_idx, container_remote_manifests_p20_pkey	1 / 1	20.59	0.129ms	49 / 0	1/64, 1/64, 1/64, 1/64, 1/64
datastore.containerRemoteBumpManifestDownloadedAtStmt.DigestKind	Update	unique_container_remote_manifests_ns_id_image_id_digest (child …_p6…_idx2)	1 / 1	12.27	0.123ms	46 / 0	1/64, 1/64, 1/64, 1/64
datastore.containerRemoteBumpImageDownloadedAtStmt.ManifestRoute	Update	n/a at 50 rows; container_remote_images_p63_pkey at 5000	0 / 0	1.75	0.079ms	30 / 0	1/64
datastore.containerRemoteBumpImageDownloadedAtStmt.BlobRoute	Update	container_remote_blobs_p58_namespace_id_container_remote_im_idx	0 / 0	14.02	0.088ms	36 / 0	1/64, 1/64, 1/64, 1/64

Every partitioned table every statement touches prunes to exactly one of its 64 hash partitions, so no statement is flagged for fan-out. The Partitions cells list one scanned/modulus pair per partitioned table in the plan, in the order the plan reaches them: for the tag route container_remote_manifests, container_remote_tags, container_remote_images, container_remote_repositories, repositories; the digest route drops the tag entry; the blob route reads container_remote_blobs in place of it; the manifest-route image bump touches container_remote_images alone.

Both image bumps report 0 / 0 at the root because an UPDATE with no RETURNING projects no rows; the scan feeding each matched 1 / 1. The manifest bump reports 1 / 1 because it does carry a RETURNING.

Query notes:

• datastore.containerRemoteBumpImageDownloadedAtStmt.ManifestRoute: the primary run seeds container_remote_images at 50 rows, which is the write-target sizing, and at that size the planner correctly prefers a Seq Scan over the 50-row partition. That says nothing about the access path at scale, so this statement was also run with container_remote_images at 5000 rows in the one partition: it switches to an Index Scan on container_remote_images_p63_pkey ((id, namespace_id)), cost 8.31, execution 1.870ms. The same supplementary run of the blob route reaches its two image references through unique_container_remote_images_ns_id_repo_id_name and container_remote_images_p21_pkey. Both supplementary plans still prune to one partition. The 5000-row plans are the ones to read for access path; the table above reports the recipe's sizing.
• No anomaly from the skill's catalog fired. The blob route's IN (…) subquery is a key lookup rather than an unbounded read: it binds namespace_id and digest and resolves its image through unique_container_remote_images_ns_id_repo_id_name, against unique_container_remote_blobs_ns_id_image_id_digest on (namespace_id, container_remote_image_id, digest) WHERE soft_deleted_at IS NULL, so it yields at most one row and needs no LIMIT.

datastore.containerRemoteBumpManifestDownloadedAtStmt.TagKind

Summary: Plan matches the method's intent. The tag row is found on (namespace_id, container_remote_image_id, name), the manifest it names is fetched by primary key, and the throttle plus both soft-delete arms are applied as a Filter on that fetch. Every one of the five partitioned tables prunes to a single partition on the namespace literal, which is the property the correlated EXISTS could lose. No anomalies.

Seed shape: namespaces=1, repositories=1, container_remote_repositories=1, container_remote_images=50, blob_storage_blobs=1, blob_storage_attachments=1, container_remote_manifests=5000, container_remote_tags=5000, container_remote_blobs=5000

Rendered SQL:

UPDATE public.container_remote_manifests
SET last_downloaded_at = GREATEST(container_remote_manifests.last_downloaded_at, NOW())
WHERE (((container_remote_manifests.namespace_id = $1::uuid) AND (container_remote_manifests.soft_deleted_at IS NULL)) AND (EXISTS (
           SELECT container_remote_tags.container_remote_manifest_id AS "container_remote_tags.container_remote_manifest_id"
           FROM public.container_remote_tags
                INNER JOIN public.container_remote_images ON ((container_remote_images.id = container_remote_tags.container_remote_image_id) AND (container_remote_images.namespace_id = container_remote_tags.namespace_id))
                INNER JOIN public.container_remote_repositories ON ((container_remote_repositories.id = container_remote_images.container_remote_repository_id) AND (container_remote_repositories.namespace_id = container_remote_images.namespace_id))
                INNER JOIN public.repositories ON ((repositories.id = container_remote_repositories.repository_id) AND (repositories.namespace_id = container_remote_repositories.namespace_id))
           WHERE (((container_remote_tags.namespace_id = $2::uuid) AND (container_remote_tags.name = $3::text)) AND ((((container_remote_manifests.namespace_id = $4::uuid) AND (container_remote_manifests.id = container_remote_tags.container_remote_manifest_id)) AND (container_remote_manifests.container_remote_image_id = container_remote_tags.container_remote_image_id)) AND (container_remote_manifests.soft_deleted_at IS NULL))) AND (((((((container_remote_images.namespace_id = $5::uuid) AND (container_remote_images.container_remote_repository_id = $6::uuid)) AND (container_remote_images.name = $7::text)) AND (container_remote_images.soft_deleted_at IS NULL)) AND (container_remote_repositories.namespace_id = $8::uuid)) AND (container_remote_repositories.id = $9::uuid)) AND ((repositories.namespace_id = $10::uuid) AND (((repositories.format IN ($11, $12)) AND (repositories.kind = $13)) AND (repositories.soft_deleted_at IS NULL))))
      ))) AND ((container_remote_manifests.last_downloaded_at IS NULL) OR (container_remote_manifests.last_downloaded_at <= (NOW() - INTERVAL '1 HOUR')))
RETURNING container_remote_manifests.container_remote_image_id AS "container_remote_manifests.container_remote_image_id";

Bound args: [ns, ns, 'review-prep-tag-000001', ns, ns, crr_id, 'library/nginx', ns, crr_id, ns, 0, 3, 2] — ns is the seeded namespace uuid, crr_id the seeded container_remote_repositories.id; 0, 3 are the container-family repository formats (Docker, OCI) and 2 is kind = remote

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Update on container_remote_manifests  (cost=12.53..20.59 rows=1 width=58) (actual time=0.128..0.129 rows=1 loops=1)
  Update on container_remote_manifests_p20 container_remote_manifests_1
  Buffers: shared hit=49
  ->  Nested Loop  (cost=12.53..20.59 rows=1 width=58) (actual time=0.028..0.029 rows=1 loops=1)
        Join Filter: (container_remote_images.id = container_remote_manifests_1.container_remote_image_id)
        Buffers: shared hit=9
        ->  HashAggregate  (cost=12.25..12.26 rows=1 width=88) (actual time=0.021..0.022 rows=1 loops=1)
              Group Key: container_remote_tags.container_remote_manifest_id, container_remote_tags.container_remote_image_id
              Batches: 1  Memory Usage: 24kB
              Buffers: shared hit=6
              ->  Nested Loop  (cost=0.28..12.24 rows=1 width=88) (actual time=0.016..0.019 rows=1 loops=1)
                    Join Filter: (container_remote_repositories.repository_id = repositories.id)
                    Buffers: shared hit=6
                    ->  Nested Loop  (cost=0.28..11.21 rows=1 width=110) (actual time=0.014..0.016 rows=1 loops=1)
                          Buffers: shared hit=5
                          ->  Nested Loop  (cost=0.28..10.19 rows=1 width=100) (actual time=0.011..0.014 rows=1 loops=1)
                                Buffers: shared hit=4
                                ->  Seq Scan on container_remote_images_p20 container_remote_images  (cost=0.00..1.88 rows=1 width=58) (actual time=0.004..0.006 rows=1 loops=1)
                                      Filter: ((soft_deleted_at IS NULL) AND (namespace_id = 'b0d39b5c-0f7b-4b87-9842-5d03466e4aa1'::uuid) AND (container_remote_repository_id = '45ce138b-fc0c-44ca-9e46-d17358b09a8f'::uuid) AND (name = 'library/nginx'::text))
                                      Rows Removed by Filter: 49
                                      Buffers: shared hit=1
                                ->  Index Scan using container_remote_tags_p20_namespace_id_container_remote_ima_idx on container_remote_tags_p20 container_remote_tags  (cost=0.28..8.30 rows=1 width=58) (actual time=0.007..0.007 rows=1 loops=1)
                                      Index Cond: ((namespace_id = 'b0d39b5c-0f7b-4b87-9842-5d03466e4aa1'::uuid) AND (container_remote_image_id = container_remote_images.id) AND (name = 'review-prep-tag-000001'::text))
                                      Buffers: shared hit=3
                          ->  Seq Scan on container_remote_repositories_p20 container_remote_repositories  (cost=0.00..1.01 rows=1 width=58) (actual time=0.002..0.002 rows=1 loops=1)
                                Filter: ((namespace_id = 'b0d39b5c-0f7b-4b87-9842-5d03466e4aa1'::uuid) AND (id = '45ce138b-fc0c-44ca-9e46-d17358b09a8f'::uuid))
                                Buffers: shared hit=1
                    ->  Seq Scan on repositories_p20 repositories  (cost=0.00..1.02 rows=1 width=42) (actual time=0.002..0.002 rows=1 loops=1)
                          Filter: ((soft_deleted_at IS NULL) AND (format = ANY ('{0,3}'::smallint[])) AND (namespace_id = 'b0d39b5c-0f7b-4b87-9842-5d03466e4aa1'::uuid) AND (kind = '2'::smallint))
                          Buffers: shared hit=1
        ->  Index Scan using container_remote_manifests_p20_pkey on container_remote_manifests_p20 container_remote_manifests_1  (cost=0.28..8.31 rows=1 width=50) (actual time=0.006..0.006 rows=1 loops=1)
              Index Cond: ((id = container_remote_tags.container_remote_manifest_id) AND (namespace_id = 'b0d39b5c-0f7b-4b87-9842-5d03466e4aa1'::uuid))
              Filter: ((soft_deleted_at IS NULL) AND (soft_deleted_at IS NULL) AND (container_remote_tags.container_remote_image_id = container_remote_image_id) AND ((last_downloaded_at IS NULL) OR (last_downloaded_at <= (now() - '01:00:00'::interval))))
              Buffers: shared hit=3
Planning:
  Buffers: shared hit=598
Planning Time: 1.746 ms
Trigger for constraint fk_container_remote_manifests_blob_storage_attachment_id_bsa on container_remote_manifests_p20: time=0.056 calls=1
Trigger for constraint fk_container_remote_manifests_container_remote_image_id on container_remote_manifests_p20: time=1.201 calls=1
Trigger for constraint fk_container_remote_manifests_namespace_id_namespaces on container_remote_manifests_p20: time=0.014 calls=1
Trigger for constraint fk_container_remote_manifests_ns_id_blob_sha256_blobs on container_remote_manifests_p20: time=0.033 calls=1
Execution Time: 1.573 ms

Timings: planning 1.746ms, execution 1.573ms, total 3.319ms.

datastore.containerRemoteBumpManifestDownloadedAtStmt.DigestKind

Summary: Plan matches the method's intent. The manifest is reached directly through the (namespace_id, container_remote_image_id, digest) partial unique index, with the image supplied by the EXISTS and the throttle applied as a Filter. Four partitioned tables, each pruned to one partition. The index name in the plan reads container_remote_manifests_p6…_idx2 — that is the child index on partition _p62, truncated at the 63-character identifier limit with a collision suffix, not a second partition. No anomalies.

Seed shape: namespaces=1, repositories=1, container_remote_repositories=1, container_remote_images=50, blob_storage_blobs=1, blob_storage_attachments=1, container_remote_manifests=5000, container_remote_tags=5000, container_remote_blobs=5000

Rendered SQL:

UPDATE public.container_remote_manifests
SET last_downloaded_at = GREATEST(container_remote_manifests.last_downloaded_at, NOW())
WHERE (((container_remote_manifests.namespace_id = $1::uuid) AND (container_remote_manifests.soft_deleted_at IS NULL)) AND ((container_remote_manifests.digest = $2::bytea) AND (EXISTS (
           SELECT container_remote_images.id AS "container_remote_images.id"
           FROM public.container_remote_images
                INNER JOIN public.container_remote_repositories ON ((container_remote_repositories.id = container_remote_images.container_remote_repository_id) AND (container_remote_repositories.namespace_id = container_remote_images.namespace_id))
                INNER JOIN public.repositories ON ((repositories.id = container_remote_repositories.repository_id) AND (repositories.namespace_id = container_remote_repositories.namespace_id))
           WHERE ((container_remote_images.id = container_remote_manifests.container_remote_image_id) AND (container_remote_images.namespace_id = container_remote_manifests.namespace_id)) AND (((((((container_remote_images.namespace_id = $3::uuid) AND (container_remote_images.container_remote_repository_id = $4::uuid)) AND (container_remote_images.name = $5::text)) AND (container_remote_images.soft_deleted_at IS NULL)) AND (container_remote_repositories.namespace_id = $6::uuid)) AND (container_remote_repositories.id = $7::uuid)) AND ((repositories.namespace_id = $8::uuid) AND (((repositories.format IN ($9, $10)) AND (repositories.kind = $11)) AND (repositories.soft_deleted_at IS NULL))))
      )))) AND ((container_remote_manifests.last_downloaded_at IS NULL) OR (container_remote_manifests.last_downloaded_at <= (NOW() - INTERVAL '1 HOUR')))
RETURNING container_remote_manifests.container_remote_image_id AS "container_remote_manifests.container_remote_image_id";

Bound args: [ns, \x00…01 (32-byte digest), ns, crr_id, 'library/nginx', ns, crr_id, ns, 0, 3, 2] — the digest is one of the 5000 seeded manifest digests

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Update on container_remote_manifests  (cost=4.22..12.27 rows=1 width=48) (actual time=0.122..0.123 rows=1 loops=1)
  Update on container_remote_manifests_p62 container_remote_manifests_1
  Buffers: shared hit=46
  ->  Nested Loop  (cost=4.22..12.27 rows=1 width=48) (actual time=0.034..0.035 rows=1 loops=1)
        Buffers: shared hit=6
        ->  HashAggregate  (cost=3.93..3.94 rows=1 width=62) (actual time=0.012..0.013 rows=1 loops=1)
              Group Key: container_remote_images.id, container_remote_images.namespace_id
              Batches: 1  Memory Usage: 24kB
              Buffers: shared hit=3
              ->  Nested Loop  (cost=0.00..3.93 rows=1 width=62) (actual time=0.008..0.010 rows=1 loops=1)
                    Join Filter: (container_remote_repositories.repository_id = repositories.id)
                    Buffers: shared hit=3
                    ->  Nested Loop  (cost=0.00..2.90 rows=1 width=84) (actual time=0.006..0.008 rows=1 loops=1)
                          Buffers: shared hit=2
                          ->  Seq Scan on container_remote_images_p62 container_remote_images  (cost=0.00..1.88 rows=1 width=58) (actual time=0.004..0.005 rows=1 loops=1)
                                Filter: ((soft_deleted_at IS NULL) AND (container_remote_repository_id = '2f151dc7-a5f4-42e9-884c-494179622710'::uuid) AND (namespace_id = '347b1513-231e-49f0-92ec-ccd55d8ece88'::uuid) AND (name = 'library/nginx'::text))
                                Rows Removed by Filter: 49
                                Buffers: shared hit=1
                          ->  Seq Scan on container_remote_repositories_p62 container_remote_repositories  (cost=0.00..1.01 rows=1 width=58) (actual time=0.002..0.002 rows=1 loops=1)
                                Filter: ((id = '2f151dc7-a5f4-42e9-884c-494179622710'::uuid) AND (namespace_id = '347b1513-231e-49f0-92ec-ccd55d8ece88'::uuid))
                                Buffers: shared hit=1
                    ->  Seq Scan on repositories_p62 repositories  (cost=0.00..1.02 rows=1 width=42) (actual time=0.002..0.002 rows=1 loops=1)
                          Filter: ((soft_deleted_at IS NULL) AND (format = ANY ('{0,3}'::smallint[])) AND (namespace_id = '347b1513-231e-49f0-92ec-ccd55d8ece88'::uuid) AND (kind = '2'::smallint))
                          Buffers: shared hit=1
        ->  Index Scan using container_remote_manifests_p6_namespace_id_container_remot_idx2 on container_remote_manifests_p62 container_remote_manifests_1  (cost=0.28..8.31 rows=1 width=50) (actual time=0.008..0.008 rows=1 loops=1)
              Index Cond: ((namespace_id = '347b1513-231e-49f0-92ec-ccd55d8ece88'::uuid) AND (container_remote_image_id = container_remote_images.id) AND (digest = '\x0000000000000000000000000000000000000000000000000000000000000001'::bytea))
              Filter: ((soft_deleted_at IS NULL) AND ((last_downloaded_at IS NULL) OR (last_downloaded_at <= (now() - '01:00:00'::interval))))
              Buffers: shared hit=3
Planning:
  Buffers: shared hit=468
Planning Time: 1.220 ms
Trigger for constraint fk_container_remote_manifests_blob_storage_attachment_id_bsa on container_remote_manifests_p62: time=0.056 calls=1
Trigger for constraint fk_container_remote_manifests_container_remote_image_id on container_remote_manifests_p62: time=1.215 calls=1
Trigger for constraint fk_container_remote_manifests_namespace_id_namespaces on container_remote_manifests_p62: time=0.009 calls=1
Trigger for constraint fk_container_remote_manifests_ns_id_blob_sha256_blobs on container_remote_manifests_p62: time=0.032 calls=1
Execution Time: 1.555 ms

Timings: planning 1.220ms, execution 1.555ms, total 2.775ms.

datastore.containerRemoteBumpImageDownloadedAtStmt.ManifestRoute

Summary: Plan matches the method's intent: a single-partition update of one image row keyed on (namespace_id, id), the id being the one the manifest bump's RETURNING produced. At the 50-row write-target sizing the planner picks a Seq Scan over the partition, which is correct at that size and is a seed artifact rather than a finding — see the Query notes above for the 5000-row run, where it switches to container_remote_images_p63_pkey. The soft-delete arm shows as a Filter, which is what turns a raced eviction into zero rows instead of a write to the tombstone.

Seed shape: namespaces=1, repositories=1, container_remote_repositories=1, container_remote_images=50, blob_storage_blobs=1, blob_storage_attachments=1, container_remote_manifests=5000, container_remote_tags=5000, container_remote_blobs=5000

Rendered SQL:

UPDATE public.container_remote_images
SET last_downloaded_at = GREATEST(container_remote_images.last_downloaded_at, NOW())
WHERE ((container_remote_images.namespace_id = $1::uuid) AND (container_remote_images.soft_deleted_at IS NULL)) AND (container_remote_images.id = $2::uuid);

Bound args: [ns, image_id] — image_id is the seeded container_remote_images.id the manifests hang off

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Update on container_remote_images  (cost=0.00..1.75 rows=0 width=0) (actual time=0.079..0.079 rows=0 loops=1)
  Update on container_remote_images_p50 container_remote_images_1
  Buffers: shared hit=30
  ->  Seq Scan on container_remote_images_p50 container_remote_images_1  (cost=0.00..1.75 rows=1 width=18) (actual time=0.004..0.005 rows=1 loops=1)
        Filter: ((soft_deleted_at IS NULL) AND (namespace_id = '1f76ba47-e6a1-4a0a-84f8-54581a3cd7be'::uuid) AND (id = '42c02a06-45fc-4b79-9d9a-054d114dfda6'::uuid))
        Rows Removed by Filter: 49
        Buffers: shared hit=1
Planning:
  Buffers: shared hit=59
Planning Time: 0.290 ms
Trigger for constraint fk_container_remote_images_container_remote_repository_id on container_remote_images_p50: time=1.720 calls=1
Trigger for constraint fk_container_remote_images_namespace_id_namespaces on container_remote_images_p50: time=0.010 calls=1
Execution Time: 1.914 ms

Timings: planning 0.290ms, execution 1.914ms, total 2.204ms.

datastore.containerRemoteBumpImageDownloadedAtStmt.BlobRoute

Summary: Plan matches the method's intent. The blob is found on (namespace_id, container_remote_image_id, digest), its parent image id is projected, and the enclosing UPDATE keys on that id with its own namespace and soft-delete arms applied. Four partitioned tables, each pruned to one partition — including the namespace equality stated on the UPDATE itself, which is what prunes the target rather than leaving the planner to propagate one out of the subquery. No anomalies.

Seed shape: namespaces=1, repositories=1, container_remote_repositories=1, container_remote_images=50, blob_storage_blobs=1, blob_storage_attachments=1, container_remote_manifests=5000, container_remote_tags=5000, container_remote_blobs=5000

Rendered SQL:

UPDATE public.container_remote_images
SET last_downloaded_at = GREATEST(container_remote_images.last_downloaded_at, NOW())
WHERE ((container_remote_images.namespace_id = $1::uuid) AND (container_remote_images.soft_deleted_at IS NULL)) AND (container_remote_images.id IN ((
           SELECT container_remote_blobs.container_remote_image_id AS "container_remote_blobs.container_remote_image_id"
           FROM public.container_remote_blobs
                INNER JOIN public.container_remote_images ON ((container_remote_images.id = container_remote_blobs.container_remote_image_id) AND (container_remote_images.namespace_id = container_remote_blobs.namespace_id))
                INNER JOIN public.container_remote_repositories ON ((container_remote_repositories.id = container_remote_images.container_remote_repository_id) AND (container_remote_repositories.namespace_id = container_remote_images.namespace_id))
                INNER JOIN public.repositories ON ((repositories.id = container_remote_repositories.repository_id) AND (repositories.namespace_id = container_remote_repositories.namespace_id))
           WHERE (((container_remote_blobs.namespace_id = $2::uuid) AND (container_remote_blobs.digest = $3::bytea)) AND (container_remote_blobs.soft_deleted_at IS NULL)) AND (((((((container_remote_images.namespace_id = $4::uuid) AND (container_remote_images.container_remote_repository_id = $5::uuid)) AND (container_remote_images.name = $6::text)) AND (container_remote_images.soft_deleted_at IS NULL)) AND (container_remote_repositories.namespace_id = $7::uuid)) AND (container_remote_repositories.id = $8::uuid)) AND ((repositories.namespace_id = $9::uuid) AND (((repositories.format IN ($10, $11)) AND (repositories.kind = $12)) AND (repositories.soft_deleted_at IS NULL))))
      )));

Bound args: [ns, ns, \x00…01 (32-byte digest), ns, crr_id, 'library/nginx', ns, crr_id, ns, 0, 3, 2] — the digest is one of the 5000 seeded blob digests

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Update on container_remote_images  (cost=12.25..14.02 rows=0 width=0) (actual time=0.088..0.088 rows=0 loops=1)
  Update on container_remote_images_p58 container_remote_images_1
  Buffers: shared hit=36
  ->  Nested Loop  (cost=12.25..14.02 rows=1 width=58) (actual time=0.020..0.021 rows=1 loops=1)
        Buffers: shared hit=7
        ->  HashAggregate  (cost=12.25..12.26 rows=1 width=72) (actual time=0.018..0.019 rows=1 loops=1)
              Group Key: container_remote_blobs.container_remote_image_id
              Batches: 1  Memory Usage: 24kB
              Buffers: shared hit=6
              ->  Nested Loop  (cost=0.28..12.24 rows=1 width=72) (actual time=0.013..0.016 rows=1 loops=1)
                    Join Filter: (container_remote_repositories.repository_id = repositories.id)
                    Buffers: shared hit=6
                    ->  Nested Loop  (cost=0.28..11.21 rows=1 width=94) (actual time=0.011..0.013 rows=1 loops=1)
                          Buffers: shared hit=5
                          ->  Nested Loop  (cost=0.28..10.19 rows=1 width=84) (actual time=0.009..0.011 rows=1 loops=1)
                                Buffers: shared hit=4
                                ->  Seq Scan on container_remote_images_p58 container_remote_images_2  (cost=0.00..1.88 rows=1 width=58) (actual time=0.003..0.005 rows=1 loops=1)
                                      Filter: ((soft_deleted_at IS NULL) AND (namespace_id = '06d0ffc1-f5e6-41af-8f54-c9cfca94c082'::uuid) AND (container_remote_repository_id = '9e86fde1-bee5-4c8c-be49-865d4fb0a205'::uuid) AND (name = 'library/nginx'::text))
                                      Rows Removed by Filter: 49
                                      Buffers: shared hit=1
                                ->  Index Scan using container_remote_blobs_p58_namespace_id_container_remote_im_idx on container_remote_blobs_p58 container_remote_blobs  (cost=0.28..8.30 rows=1 width=42) (actual time=0.005..0.005 rows=1 loops=1)
                                      Index Cond: ((namespace_id = '06d0ffc1-f5e6-41af-8f54-c9cfca94c082'::uuid) AND (container_remote_image_id = container_remote_images_2.id) AND (digest = '\x0000000000000000000000000000000000000000000000000000000000000001'::bytea))
                                      Filter: (soft_deleted_at IS NULL)
                                      Buffers: shared hit=3
                          ->  Seq Scan on container_remote_repositories_p58 container_remote_repositories  (cost=0.00..1.01 rows=1 width=58) (actual time=0.001..0.001 rows=1 loops=1)
                                Filter: ((namespace_id = '06d0ffc1-f5e6-41af-8f54-c9cfca94c082'::uuid) AND (id = '9e86fde1-bee5-4c8c-be49-865d4fb0a205'::uuid))
                                Buffers: shared hit=1
                    ->  Seq Scan on repositories_p58 repositories  (cost=0.00..1.02 rows=1 width=42) (actual time=0.002..0.002 rows=1 loops=1)
                          Filter: ((soft_deleted_at IS NULL) AND (format = ANY ('{0,3}'::smallint[])) AND (namespace_id = '06d0ffc1-f5e6-41af-8f54-c9cfca94c082'::uuid) AND (kind = '2'::smallint))
                          Buffers: shared hit=1
        ->  Seq Scan on container_remote_images_p58 container_remote_images_1  (cost=0.00..1.75 rows=1 width=34) (actual time=0.001..0.001 rows=1 loops=1)
              Filter: ((soft_deleted_at IS NULL) AND (namespace_id = '06d0ffc1-f5e6-41af-8f54-c9cfca94c082'::uuid) AND (container_remote_images_2.id = id))
              Buffers: shared hit=1
Planning:
  Buffers: shared hit=538
Planning Time: 1.411 ms
Trigger for constraint fk_container_remote_images_container_remote_repository_id on container_remote_images_p58: time=1.526 calls=1
Trigger for constraint fk_container_remote_images_namespace_id_namespaces on container_remote_images_p58: time=0.009 calls=1
Execution Time: 1.744 ms

Timings: planning 1.411ms, execution 1.744ms, total 3.155ms.

supplementary: manifest route image bump with container_remote_images at 5000 rows

Why: the primary run seeds container_remote_images at the 50-row write-target sizing, where a Seq Scan is the planner's correct choice and the access path is therefore untested. This run is identical except that the one partition holds 5000 images.

Seed shape: as above, with container_remote_images=5000.

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Update on container_remote_images  (cost=0.28..8.31 rows=0 width=0) (actual time=0.103..0.103 rows=0 loops=1)
  Update on container_remote_images_p63 container_remote_images_1
  Buffers: shared hit=37
  ->  Index Scan using container_remote_images_p63_pkey on container_remote_images_p63 container_remote_images_1  (cost=0.28..8.31 rows=1 width=18) (actual time=0.007..0.007 rows=1 loops=1)
        Index Cond: ((id = '05530742-a884-4634-ba82-e76e38e1973a'::uuid) AND (namespace_id = 'b5487038-4b9a-4907-ac8c-c6b4f5456e3c'::uuid))
        Filter: (soft_deleted_at IS NULL)
        Buffers: shared hit=3
Planning:
  Buffers: shared hit=72
Planning Time: 0.411 ms
Trigger for constraint fk_container_remote_images_container_remote_repository_id on container_remote_images_p63: time=1.622 calls=1
Trigger for constraint fk_container_remote_images_namespace_id_namespaces on container_remote_images_p63: time=0.009 calls=1
Execution Time: 1.870 ms

supplementary: blob route image bump with container_remote_images at 5000 rows

Why: the primary run seeds container_remote_images at the 50-row write-target sizing, where a Seq Scan is the planner's correct choice and the access path is therefore untested. This run is identical except that the one partition holds 5000 images.

Seed shape: as above, with container_remote_images=5000.

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Update on container_remote_images  (cost=18.96..27.01 rows=0 width=0) (actual time=0.120..0.121 rows=0 loops=1)
  Update on container_remote_images_p21 container_remote_images_1
  Buffers: shared hit=45
  ->  Nested Loop  (cost=18.96..27.01 rows=1 width=58) (actual time=0.031..0.032 rows=1 loops=1)
        Buffers: shared hit=11
        ->  HashAggregate  (cost=18.68..18.69 rows=1 width=72) (actual time=0.024..0.025 rows=1 loops=1)
              Group Key: container_remote_blobs.container_remote_image_id
              Batches: 1  Memory Usage: 24kB
              Buffers: shared hit=8
              ->  Nested Loop  (cost=0.56..18.68 rows=1 width=72) (actual time=0.021..0.022 rows=1 loops=1)
                    Join Filter: (container_remote_repositories.repository_id = repositories.id)
                    Buffers: shared hit=8
                    ->  Nested Loop  (cost=0.56..17.65 rows=1 width=94) (actual time=0.018..0.019 rows=1 loops=1)
                          Buffers: shared hit=7
                          ->  Nested Loop  (cost=0.56..16.62 rows=1 width=84) (actual time=0.016..0.016 rows=1 loops=1)
                                Buffers: shared hit=6
                                ->  Index Scan using container_remote_images_p21_namespace_id_container_remote_r_idx on container_remote_images_p21 container_remote_images_2  (cost=0.28..8.30 rows=1 width=58) (actual time=0.008..0.008 rows=1 loops=1)
                                      Index Cond: ((namespace_id = 'b0636108-5d1b-4b2d-a0ae-8794688ad833'::uuid) AND (container_remote_repository_id = 'c616430c-68db-4155-b72a-8d259f1895b5'::uuid) AND (name = 'library/nginx'::text))
                                      Filter: (soft_deleted_at IS NULL)
                                      Buffers: shared hit=3
                                ->  Index Scan using container_remote_blobs_p21_namespace_id_container_remote_im_idx on container_remote_blobs_p21 container_remote_blobs  (cost=0.28..8.30 rows=1 width=42) (actual time=0.007..0.007 rows=1 loops=1)
                                      Index Cond: ((namespace_id = 'b0636108-5d1b-4b2d-a0ae-8794688ad833'::uuid) AND (container_remote_image_id = container_remote_images_2.id) AND (digest = '\x0000000000000000000000000000000000000000000000000000000000000001'::bytea))
                                      Filter: (soft_deleted_at IS NULL)
                                      Buffers: shared hit=3
                          ->  Seq Scan on container_remote_repositories_p21 container_remote_repositories  (cost=0.00..1.01 rows=1 width=58) (actual time=0.002..0.002 rows=1 loops=1)
                                Filter: ((namespace_id = 'b0636108-5d1b-4b2d-a0ae-8794688ad833'::uuid) AND (id = 'c616430c-68db-4155-b72a-8d259f1895b5'::uuid))
                                Buffers: shared hit=1
                    ->  Seq Scan on repositories_p21 repositories  (cost=0.00..1.02 rows=1 width=42) (actual time=0.002..0.002 rows=1 loops=1)
                          Filter: ((soft_deleted_at IS NULL) AND (format = ANY ('{0,3}'::smallint[])) AND (namespace_id = 'b0636108-5d1b-4b2d-a0ae-8794688ad833'::uuid) AND (kind = '2'::smallint))
                          Buffers: shared hit=1
        ->  Index Scan using container_remote_images_p21_pkey on container_remote_images_p21 container_remote_images_1  (cost=0.28..8.30 rows=1 width=34) (actual time=0.005..0.006 rows=1 loops=1)
              Index Cond: ((id = container_remote_images_2.id) AND (namespace_id = 'b0636108-5d1b-4b2d-a0ae-8794688ad833'::uuid))
              Filter: (soft_deleted_at IS NULL)
              Buffers: shared hit=3
Planning:
  Buffers: shared hit=537
Planning Time: 1.489 ms
Trigger for constraint fk_container_remote_images_container_remote_repository_id on container_remote_images_p21: time=1.643 calls=1
Trigger for constraint fk_container_remote_images_namespace_id_namespaces on container_remote_images_p21: time=0.011 calls=1
Execution Time: 1.954 ms

Related to #288