feat(datastore): apply the download count as a guarded batch part 1/3 (repository-column-writers plan: 4/5) (!2494) · Merge requests · GitLab.org / ops / Artifact Registry

Part 1 of 3 of the repository download-count family. This part is the SQL half and nothing reaches it on merge: internal/datastore imports neither counterbuf nor accounting, so the delta type, the two statements and their tests stand on their own.

What this adds

RepoDownloadsDelta, which carries no Baseline. No reconciliation pass recomputes repositories.downloads_count, so a guard on last_reconciled_at would skip a scope no pass had recomputed and drop the delta the same transaction stops staging. soft_deleted_at IS NULL stands in its place on both statements, because a tombstone can land between the baseline read and the apply and a download must not be counted onto one. DeltaDownloads must not be negative, and orderedDownloadsDeltas refuses one before the apply builds a statement argument. Nothing decrements this column: the three production writers are Maven's BumpRepoDownloadCount (+1), IncrementRepositoryDownloadsCount (+n, already refusing a non-positive n) and this apply, all additive. So a negative is a caller's defect, and with no pass to recompute the column it would sit below zero for good, served as stored by the management API and summed into the namespace rollup. A batch carrying one fails whole and moves no scope in that chunk; that is the trade, a retried and logged chunk against a permanent negative. A zero is still accepted, since it adds nothing. No CHECK (>= 0) is added, because a constraint would also refuse the positive delta that is the only way to repair a row already below zero.
DownloadsBaselines and ApplyDownloadsDeltas. The apply reuses lockRepoScopesStmt itself rather than a copy, so a downloads chunk and a repository-scoped chunk take the same repositories rows in the same order and cannot deadlock against each other.
IncrementRepositoryDownloadsCount, the single-row writer. Unreachable until a per-read caller exists; the drain will commit its chunks through ApplyDownloadsDeltas instead.
Five corrections in docs/dev/storage-accounting.md, whose claims this part falsifies, plus one addition. The counter-drain section counted six statements at :940 and :969 where counter_drain.go now owns eight, and the rule at :974 prohibited any filter on repositories.soft_deleted_at, which is exactly what downloadsBaselinesStmt and applyDownloadsDeltasStmt do. The prohibition is now scoped to the six statements it covers, and the downloads family has its own clause naming both statements and why their reason differs. The other two carve this family out of the drain's self-heal claim. The flush-safe merge closed on "All three self-heal on the next pass" (:887, :894 on the branch), and ## Error cases restated the same fact in its "Two chunks run the same scope concurrently" row; both hold for the four reconciled counters, which reconciliation overwrites from source, and both are false for a column no pass recomputes. Each now scopes the claim to those four and states the asymmetry: a negative delta is refused before it reaches a statement, by orderedDownloadsDeltas, while a positive over-count has neither a guard at batch entry nor a reconciliation backstop. None of the five was in the plan's Step 4 sweep entry, so nothing later re-fixed them; !2498 (merged), the batch docs(plans) MR for this step, is where the two self-heal sites get their sweep-entry row, because guardrail 4 bars this MR from editing the plan file. The addition prices the downloads apply's write amplification, which no doc carried for this family: 11 index tuples per drained row version for a live hosted or virtual repository and 12 for a live remote one, the coalescing that makes that cost per scope per tick rather than per download, and the autovacuum_vacuum_scale_factor lever.

What this part does not reach

Nothing enqueues or calls any of it. There is no family constant, no dirty set, no worker and no producer until parts 2 and 3, so merging this alone changes no behaviour and moves no column.

The column itself has no reconciliation backstop, in this part or after the whole step: every shed, lost reply and over-count on this family is permanent rather than bounded. Part 3 documents that across the accounting surfaces.

The negative-delta guard does not reach a row that already sits below zero. It stops a new negative from landing and heals nothing stored, and no pass will. No production path can have written such a row, since all three writers of the column are additive, so the residual is what a future decrementing writer would leave, and that would need a backfill rather than this guard.

Stack

Merge in this order. Each part builds and tests on its own, and the family is inert until part 3. Each part is hard-gated behind its parent, so none can merge early.

#	MR	What it adds	Reviewable LOC
1	!2494 (merged)	The SQL half: the delta type, the two statements, the single-row writer, and the `storage-accounting.md` claims this part falsifies plus the drain's write-amplification pricing	1683
2	!2495 (merged)	The Redis half, `Emitter.EmitRepoDownloads`, the chunk worker	1430
3	!2496 (merged)	The drain trigger, the registration, npm's producer, every doc surface	1338

Diff size

Guardrail 18 asks for a split or a justification past 500 reviewable LOC. This step was split three ways and each part is still over it, so the justification stands per part rather than being cleared by the split.

By file group, re-derived at ca65e85e4 with git diff --numstat origin/main...HEAD: Go tests 7 files +1172/-125 = 1297, Go production 3 files +265/-83 = 348, docs 1 file +31/-7 = 38. Tests are 1297 of the 1683 reviewable LOC, and the indivisible unit is counter_drain.go with the 1127 LOC of unit and integration tests that cover its two new statements: the plan cases, the guard and handle rejections, the oversized batch, the duplicate scope, the lock hold, the tombstone race, the coalescing property above a delta of one, the refused negative delta, the per-family baseline wrap, and the columns the downloads apply must not move. Splitting those from the statements they cover would leave either half unreviewable.

Testing

internal/datastore unit and integration suites. No format protocol behaviour changes, so guardrail 12's conformance suites do not apply, and no e2e scenario is added or affected here: the catalog row for the delivered tarball's increment lands in part 3 with the producer it describes.

Database Review Evidence

Queries

Note

Plans are from EXPLAIN (ANALYZE, BUFFERS) against an ephemeral PostgreSQL 17 container (matching GL_PG_CURR_VERSION from .gitlab-ci-other-versions.yml), with synthesized seed data rolled back per query and the container torn down at the end of the run. Numbers reflect moderate cardinality and do not capture production-scale effects. See Database review evidence for seed sizing, methodology, and the anomalies the skill flags. Expand each row's details for the seed shape, rendered SQL, bound args, and raw plan.

Two deviations from the default recipe, both stated so a reader can re-run them. The write targets are seeded at the read-target size (5000 rows) rather than the recipe's 50, because all four statements hit one hash-partitioned table and a 50-row repositories plans as a sequential scan whatever the statement's shape. And the plan text is reproduced with the 63 sibling partition nodes that return no rows collapsed into one marker line, and the 500-element uuid[] bind literals shortened to their first value plus a count: verbatim, the four plans come to 3.5 MB.

Method	Plan node	Index	Rows (plan / actual)	Cost	Time	Buffers (hit / read)	Partitions
`datastore.CounterDrainStore.ApplyDownloadsDeltas.apply`	Update	`repositories_p25_pkey`	0 / 0	248.90	9.671ms	14435 / 0	⚠️ 64/64
`datastore.CounterDrainStore.ApplyDownloadsDeltas.lock`	LockRows	`repositories_p49_pkey`	141 / 500	256.80	0.901ms	592 / 0	⚠️ 64/64
`datastore.CounterDrainStore.DownloadsBaselines`	Hash Join	`repositories_p03_pkey`	2 / 500	249.25	0.370ms	92 / 0	⚠️ 64/64
`datastore.RepositoryStore.IncrementRepositoryDownloadsCount`	Update	`repositories_p52_pkey`	0 / 0	8.30	0.153ms	65 / 0	1/64

Warning

Partition fan-out. These statements scan more than one partition of a hash-partitioned table:

datastore.CounterDrainStore.ApplyDownloadsDeltas.apply: 64/64 partitions of repositories, and 64 per-partition Update targets. The partition key reaches the statement only through unnest($1::uuid[], ...). PostgreSQL prunes a hash partition only from a predicate comparing the key to a value it knows at planning or at execution start, not from a join qual, so every partition is opened however few repositories the chunk names. The fan-out stands because it is the shape the merged repository-scoped family already has: measured in the same container, applyRepoDeltasStmt on main is 64/64 scan and 64/64 update at 14518 buffers and 14.314ms, against this statement's 14435 and 11.823ms. Closing it means chunking the drain by namespace so r.namespace_id = $k binds a single value, which would change both families and belongs in its own change.
datastore.CounterDrainStore.ApplyDownloadsDeltas.lock: 64/64 partitions of repositories. Same cause, and the statement is lockRepoScopesStmt itself rather than a copy, so this row measures a statement already on main.
datastore.CounterDrainStore.DownloadsBaselines: 64/64 partitions of repositories. Same cause; repoBaselinesStmt on main measures 64/64 too, with the same Hash Join over an Append of all 64.

Query notes:

datastore.CounterDrainStore.DownloadsBaselines and datastore.CounterDrainStore.ApplyDownloadsDeltas.apply: plan-versus-actual row mismatch of 250x on the driving join (2 estimated, 500 returned). PostgreSQL has no statistics for an unnest() result against a partitioned target and assumes about one row per partition. repoBaselinesStmt on main estimates 2 against the same 500, so the estimate is a property of the family's shape rather than of these statements. It costs nothing at this batch size, because the chosen Hash Join is what the true cardinality wants anyway.
The repeated key restriction AND r.id = ANY($2::uuid[]) was measured against a copy with the clause removed. With it, the partition holding the batch is reached by Bitmap Index Scan on repositories_p03_pkey and 500 rows enter the hash. Without it, the same partition is read by Seq Scan and all 5000 rows enter the hash. The partition count is 64 either way, so the clause buys the index path, not pruning.

datastore.CounterDrainStore.ApplyDownloadsDeltas.apply

Summary: Plan matches the method's intent. The chunk's rows are found through the partition's primary key, soft_deleted_at IS NULL is applied as a filter on the same scan, and the update lands on the one partition that holds them. What the plan also shows is that all 64 partitions are opened, as scan children and again as Update targets, which is where the root node's 14435 buffer hits come from rather than from the 500 rows moved. The shape is inherited from applyRepoDeltasStmt, measured at 14518 buffers in the same container.

Seed shape: namespaces=1, repositories=5000

Rendered SQL:

UPDATE repositories AS r
		SET downloads_count = r.downloads_count + v.delta_downloads
		FROM unnest($1::uuid[], $2::uuid[], $3::bigint[])
		     AS v (namespace_id, id, delta_downloads)
		WHERE r.namespace_id = v.namespace_id AND r.id = v.id
		  AND r.soft_deleted_at IS NULL
		  AND r.id = ANY($2::uuid[])

Bound args: $1 = 500 copies of the seeded namespace id, $2 = the 500 seeded repository ids, $3 = 500 deltas of 1. The batch size is the storage_accounting.drain_chunk_size default of 500, under MaxCounterDrainBatchSize of 1000.

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Update on repositories r  (cost=241.27..248.90 rows=0 width=0) (actual time=9.662..9.671 rows=0 loops=1)
  Update on repositories_p00 r_1   [... and 63 further per-partition Update targets, elided ...]
  Buffers: shared hit=14435 dirtied=28 written=28
  ->  Hash Join  (cost=241.27..248.90 rows=2 width=82) (actual time=0.288..0.443 rows=500 loops=1)
        Hash Cond: ((v.namespace_id = r.namespace_id) AND (v.id = r.id))
        Buffers: shared hit=91
        ->  Function Scan on v  (cost=0.01..5.01 rows=500 width=104) (actual time=0.044..0.112 rows=500 loops=1)
        ->  Hash  (cost=232.82..232.82 rows=563 width=50) (actual time=0.240..0.248 rows=500 loops=1)
              Buckets: 1024  Batches: 1  Memory Usage: 49kB
              Buffers: shared hit=91
              ->  Append  (cost=1.25..232.82 rows=563 width=50) (actual time=0.056..0.207 rows=500 loops=1)
                    Buffers: shared hit=91
                    [... 63 sibling partition scans, each actual rows=0, elided ...]
                    ->  Bitmap Heap Scan on repositories_p25 r_26  (cost=56.75..151.25 rows=500 width=50) (actual time=0.041..0.139 rows=500 loops=1)
                          Recheck Cond: (id = ANY ('{001b53bb-22ea-7b56-b658-63a436720a54, ... 500 values ...}'::uuid[]))
                          Filter: (soft_deleted_at IS NULL)
                          Heap Blocks: exact=86
                          Buffers: shared hit=91
                          ->  Bitmap Index Scan on repositories_p25_pkey  (cost=0.00..55.38 rows=500 width=0) (actual time=0.033..0.033 rows=500 loops=1)
                                Index Cond: (id = ANY ('{001b53bb-22ea-7b56-b658-63a436720a54, ... 500 values ...}'::uuid[]))
                                Buffers: shared hit=5
Planning:
  Buffers: shared hit=1166
Planning Time: 8.500 ms
Trigger for constraint fk_repositories_namespace_id_namespaces on repositories_p25: time=1.079 calls=500
Execution Time: 11.823 ms

Timings: planning 8.500ms, execution 11.823ms, total 20.323ms.

datastore.CounterDrainStore.ApplyDownloadsDeltas.lock

Summary: Plan matches the method's intent, and it is the statement's ordering contract that the plan makes visible: LockRows sits above Sort on r.namespace_id, r.id, so the rows are locked in the stated order whatever the update below chooses. The batch reaches its rows through repositories_p49_pkey. All 64 partitions are opened for the same reason as the apply, and this row measures lockRepoScopesStmt itself, which the new method reuses rather than copies.

Seed shape: namespaces=1, repositories=5000

Rendered SQL:

SELECT r.namespace_id, r.id
		FROM repositories AS r
		WHERE (r.namespace_id, r.id) IN (
		        SELECT s.namespace_id, s.id
		        FROM unnest($1::uuid[], $2::uuid[]) AS s (namespace_id, id))
		  AND r.id = ANY($2::uuid[])
		ORDER BY r.namespace_id, r.id
		FOR NO KEY UPDATE

Bound args: $1 = 500 copies of the seeded namespace id, $2 = the 500 seeded repository ids.

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

LockRows  (cost=255.04..256.80 rows=141 width=98) (actual time=0.694..0.901 rows=500 loops=1)
  Buffers: shared hit=592
  ->  Sort  (cost=255.04..255.39 rows=141 width=98) (actual time=0.687..0.705 rows=500 loops=1)
        Sort Key: r.namespace_id, r.id
        Sort Method: quicksort  Memory: 79kB
        Buffers: shared hit=92
        ->  Hash Join  (cost=13.76..250.01 rows=141 width=98) (actual time=0.393..0.559 rows=500 loops=1)
              Hash Cond: ((r.namespace_id = s.namespace_id) AND (r.id = s.id))
              Buffers: shared hit=92
              ->  Append  (cost=1.25..232.82 rows=563 width=42) (actual time=0.086..0.200 rows=500 loops=1)
                    Buffers: shared hit=92
                    [... 63 sibling partition scans, each actual rows=0, elided ...]
                    ->  Bitmap Heap Scan on repositories_p49 r_50  (cost=56.75..151.25 rows=500 width=42) (actual time=0.048..0.128 rows=500 loops=1)
                          Recheck Cond: (id = ANY ('{0006a7ea-ff27-7f67-8cef-45fd0aa5f6a7, ... 500 values ...}'::uuid[]))
                          Heap Blocks: exact=87
                          Buffers: shared hit=92
                          ->  Bitmap Index Scan on repositories_p49_pkey  (cost=0.00..55.38 rows=500 width=0) (actual time=0.041..0.041 rows=500 loops=1)
                                Index Cond: (id = ANY ('{0006a7ea-ff27-7f67-8cef-45fd0aa5f6a7, ... 500 values ...}'::uuid[]))
                                Buffers: shared hit=5
              ->  Hash  (cost=9.51..9.51 rows=200 width=88) (actual time=0.301..0.301 rows=500 loops=1)
                    Buckets: 1024  Batches: 1  Memory Usage: 66kB
                    ->  HashAggregate  (cost=7.50..9.51 rows=200 width=88) (actual time=0.232..0.261 rows=500 loops=1)
                          Group Key: s.namespace_id, s.id
                          Batches: 1  Memory Usage: 185kB
                          ->  Function Scan on s  (cost=0.01..5.00 rows=500 width=88) (actual time=0.127..0.161 rows=500 loops=1)
Planning:
  Buffers: shared hit=1030
Planning Time: 6.637 ms
Execution Time: 1.041 ms

Timings: planning 6.637ms, execution 1.041ms, total 7.678ms.

datastore.CounterDrainStore.DownloadsBaselines

Summary: Plan matches the method's intent. The new soft_deleted_at IS NULL leg is applied as a filter on the partition's bitmap heap scan rather than costing a separate node, and the key restriction is what puts the read on repositories_p03_pkey. The read returns 500 rows for 92 buffer hits. The estimate of 2 rows against 500 actual, and the 64-partition Append, are both present in repoBaselinesStmt on main at the same numbers.

Seed shape: namespaces=1, repositories=5000

Rendered SQL:

SELECT r.namespace_id, r.id, r.last_reconciled_at
		FROM repositories AS r
		JOIN unnest($1::uuid[], $2::uuid[]) AS s (namespace_id, id)
		  ON r.namespace_id = s.namespace_id AND r.id = s.id
		WHERE r.id = ANY($2::uuid[])
		  AND r.soft_deleted_at IS NULL

Bound args: $1 = 500 copies of the seeded namespace id, $2 = the 500 seeded repository ids.

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Hash Join  (cost=241.62..249.25 rows=2 width=40) (actual time=0.307..0.370 rows=500 loops=1)
  Hash Cond: ((s.namespace_id = r.namespace_id) AND (s.id = r.id))
  Buffers: shared hit=92
  ->  Function Scan on s  (cost=0.01..5.00 rows=500 width=32) (actual time=0.030..0.047 rows=500 loops=1)
  ->  Hash  (cost=233.18..233.18 rows=563 width=40) (actual time=0.271..0.275 rows=500 loops=1)
        Buckets: 1024  Batches: 1  Memory Usage: 44kB
        Buffers: shared hit=92
        ->  Append  (cost=1.25..233.18 rows=563 width=40) (actual time=0.053..0.239 rows=500 loops=1)
              Buffers: shared hit=92
              [... 63 sibling partition scans, each actual rows=0, elided ...]
              ->  Bitmap Heap Scan on repositories_p03 r_4  (cost=57.11..151.61 rows=500 width=40) (actual time=0.048..0.171 rows=500 loops=1)
                    Recheck Cond: (id = ANY ('{00016e18-199c-75b7-a1c4-7a6da2034fd6, ... 500 values ...}'::uuid[]))
                    Filter: (soft_deleted_at IS NULL)
                    Heap Blocks: exact=87
                    Buffers: shared hit=92
                    ->  Bitmap Index Scan on repositories_p03_pkey  (cost=0.00..55.73 rows=500 width=0) (actual time=0.038..0.038 rows=500 loops=1)
                          Index Cond: (id = ANY ('{00016e18-199c-75b7-a1c4-7a6da2034fd6, ... 500 values ...}'::uuid[]))
                          Buffers: shared hit=5
Planning:
  Buffers: shared hit=4738
Planning Time: 16.888 ms
Execution Time: 0.531 ms

Timings: planning 16.888ms, execution 0.531ms, total 17.419ms.

datastore.RepositoryStore.IncrementRepositoryDownloadsCount

Summary: Plan matches the method's intent and is the one statement here that prunes. namespace_id arrives as a scalar bind, so PostgreSQL reaches a single partition and seeks repositories_p52_pkey on (id, namespace_id), with soft_deleted_at IS NULL as a filter on that scan. 3 buffer hits for the row itself, 65 at the root, and 0.153ms at 5000 seeded rows. No anomalies.

Seed shape: namespaces=1, repositories=5000

Rendered SQL:

UPDATE public.repositories
SET downloads_count = (repositories.downloads_count + $1)
WHERE ((repositories.namespace_id = $2::uuid) AND (repositories.id = $3::uuid)) AND (repositories.soft_deleted_at IS NULL);

Bound args: [1, 356721db-d99a-78f7-aa4a-9f9a469225ea, 7e3c8935-ac7a-7628-8463-317608f95534]

Plan (EXPLAIN (ANALYZE, BUFFERS) output):

Update on repositories  (cost=0.28..8.30 rows=0 width=0) (actual time=0.153..0.153 rows=0 loops=1)
  Update on repositories_p52 repositories_1
  Buffers: shared hit=65
  ->  Index Scan using repositories_p52_pkey on repositories_p52 repositories_1  (cost=0.28..8.30 rows=1 width=18) (actual time=0.006..0.006 rows=1 loops=1)
        Index Cond: ((id = '7e3c8935-ac7a-7628-8463-317608f95534'::uuid) AND (namespace_id = '356721db-d99a-78f7-aa4a-9f9a469225ea'::uuid))
        Filter: (soft_deleted_at IS NULL)
        Buffers: shared hit=3
Planning:
  Buffers: shared hit=164
Planning Time: 0.435 ms
Trigger for constraint fk_repositories_namespace_id_namespaces on repositories_p52: time=0.170 calls=1
Execution Time: 0.361 ms

Timings: planning 0.435ms, execution 0.361ms, total 0.796ms.

Related to #1223 (closed)

Edited Sep 11, 2026 by Dzmitry (Dima) Meshcharakou

feat(datastore): apply the download count as a guarded batch part 1/3 (repository-column-writers plan: 4/5)

What this adds

What this part does not reach

Stack

Diff size

Testing

Database Review Evidence

Queries

Merge request reports