feat(npm): add the bounded reads the streaming rebuild will walk
🎯 What this MR does
Adds the two bounded reads the streaming packument rebuild walks, and wires neither of them. No production behaviour changes here, exactly as the renderer in !1646 (merged) landed ahead of its caller.
Part 1 of 5 of Step 2 in the npm packument streaming generation plan.
| Part | MR | What it delivers |
|---|---|---|
| 2a · this MR | NpmVersionStringsByIDs + keysetVersionPager |
|
| 2b | next | blob writes routed through a hashing tee |
| 2c | the rebuild streams into its blob sessions | |
| 2d | session lifetime and the two-session bound | |
| 2e | fence re-checked per page |
Step 2 was one ~5,400-line change. It is split because
docs/dev/development-model.md asks for a split
or a justification, and because each part leaves main green on its own.
🧩 What is in it
datastore.NpmVersionStringsByIDs resolves version-row ids to their version
strings in one round trip. The rebuild needs it because every npm_tags row
carries an id and no version string, and a streaming render emits dist-tags
before it reads the first version row, so a tag cannot be resolved from the rows
the walk yields.
It selects only id and version, so a tagged row never pulls package_json
onto the heap. It caps its id set at the store's page cap rather than trusting
the caller, because npm.max_tags_per_package is operator-configurable with no
ceiling of its own; a caller with more ids batches.
keysetVersionPager is the production VersionPager for the renderer !1646 (merged)
landed. It walks one package's active version rows a keyset page at a time and
upholds the three clauses VersionPager states but cannot check: each row
appears once per walk, done is reported exactly at exhaustion, and the order is
total and repeatable.
Two behaviour-preserving supports go with them: bindDistTags is split out of
resolveDistTags so the tag-binding rule lives in one place, and
allNpmVersions narrows its parameter to the new npmVersionPageLister seam.
🔍 Worth a close read
The pager suite drives the walk against a page source that implements the real keyset predicate, so a stalled or mis-taken cursor shows up as a wrong row sequence rather than passing by construction.
NpmVersionStringsByIDs' EXPLAIN (ANALYZE, BUFFERS) at
npm.max_tags_per_package (1000, its default, which is also the store's
per-call cap). The fixture is one namespace holding a 25,000-version package
plus 500 soft-deleted rows on it and a second 1,000-version package, and the
1000 requested ids are deliberately mixed: 800 live rows of the target package,
100 of its soft-deleted rows, and 100 belonging to the sibling package. The
statement is the one go-jet renders, dumped from the same builder the method
uses:
Bitmap Heap Scan on npm_versions_p07 npm_versions (cost=277.09..1658.62 rows=944 width=24) (actual time=0.634..0.740 rows=800 loops=1)
Recheck Cond: ((id = ANY ('{<1000 uuids>}'::uuid[])) AND (namespace_id = '…'::uuid))
Filter: ((soft_deleted_at IS NULL) AND (npm_package_id = '…'::uuid))
Rows Removed by Filter: 200
Heap Blocks: exact=51
Buffers: shared hit=467
-> Bitmap Index Scan on npm_versions_p07_pkey (cost=0.00..274.35 rows=1000 width=0) (actual time=0.623..0.624 rows=1000 loops=1)
Index Cond: ((id = ANY ('{<1000 uuids>}'::uuid[])) AND (namespace_id = '…'::uuid))
Buffers: shared hit=416
Planning:
Buffers: shared hit=650
Planning Time: 1.456 ms
Execution Time: 0.773 msOne partition (namespace_id prunes to _p07), and the index scan takes both
primary-key columns — id = ANY over the list plus namespace_id — so the id
set is resolved entirely from pk_npm_versions. All buffer hits.
The two predicates that are not in that index, npm_package_id and
soft_deleted_at IS NULL, are a heap Filter, which is the number worth
reading: Rows Removed by Filter: 200. The bitmap heap scan fetches one tuple
per requested id, not per resolved one, so a caller that asks about ids
which are soft-deleted or owned by a sibling package still pays their heap
fetches. It is cheap — a residual filter on already-fetched tuples — and the
width=24 projection means none of those fetches detoast package_json, which
is the property the doc comment claims. But the cost scales with the size of the
id set the caller passes, not with the number of tags that resolve.
Execution is well under planning here (0.77 ms vs 1.46 ms), which is expected
with a 1000-element IN list. Figures are stable across five consecutive runs
(execution 0.77–0.90 ms; the row and buffer counts identical).
📏 Size
1068 reviewable LOC: 362 datastore (116 production, 246 tests), 675 the pager and its suite, 31 the two supports. Past the 500-line guidance, and this is already the smallest part of the split; the pager and its contract suite are one unit.
✅ Testing
- Unit,
-race, and integration suites green;golangci-lintreports 0 issues, including a--build-tags=integrationrun. - npm conformance (
mise run conformance:npm) passes on the head of this stack. - The datastore suite covers strictly more than the
NpmVersionsByPackageAndVersionstable it mirrors, with the cap boundary on the accepting side and cap+1 on the guard side.
Related to #241