feat(npm): add the bounded reads the streaming rebuild will walk

🎯 What this MR does

Adds the two bounded reads the streaming packument rebuild walks, and wires neither of them. No production behaviour changes here, exactly as the renderer in !1646 (merged) landed ahead of its caller.

Part 1 of 5 of Step 2 in the npm packument streaming generation plan.

📚 The stack:

Part MR What it delivers
2a · this MR 👉 this one NpmVersionStringsByIDs + keysetVersionPager
2b next blob writes routed through a hashing tee
2c the rebuild streams into its blob sessions
2d session lifetime and the two-session bound
2e fence re-checked per page

Step 2 was one ~5,400-line change. It is split because docs/dev/development-model.md asks for a split or a justification, and because each part leaves main green on its own.

🧩 What is in it

datastore.NpmVersionStringsByIDs resolves version-row ids to their version strings in one round trip. The rebuild needs it because every npm_tags row carries an id and no version string, and a streaming render emits dist-tags before it reads the first version row, so a tag cannot be resolved from the rows the walk yields.

It selects only id and version, so a tagged row never pulls package_json onto the heap. It caps its id set at the store's page cap rather than trusting the caller, because npm.max_tags_per_package is operator-configurable with no ceiling of its own; a caller with more ids batches.

keysetVersionPager is the production VersionPager for the renderer !1646 (merged) landed. It walks one package's active version rows a keyset page at a time and upholds the three clauses VersionPager states but cannot check: each row appears once per walk, done is reported exactly at exhaustion, and the order is total and repeatable.

Two behaviour-preserving supports go with them: bindDistTags is split out of resolveDistTags so the tag-binding rule lives in one place, and allNpmVersions narrows its parameter to the new npmVersionPageLister seam.

🔍 Worth a close read

The pager suite drives the walk against a page source that implements the real keyset predicate, so a stalled or mis-taken cursor shows up as a wrong row sequence rather than passing by construction.

NpmVersionStringsByIDs' EXPLAIN (ANALYZE, BUFFERS) at npm.max_tags_per_package (1000, its default, which is also the store's per-call cap). The fixture is one namespace holding a 25,000-version package plus 500 soft-deleted rows on it and a second 1,000-version package, and the 1000 requested ids are deliberately mixed: 800 live rows of the target package, 100 of its soft-deleted rows, and 100 belonging to the sibling package. The statement is the one go-jet renders, dumped from the same builder the method uses:

Bitmap Heap Scan on npm_versions_p07 npm_versions  (cost=277.09..1658.62 rows=944 width=24) (actual time=0.634..0.740 rows=800 loops=1)
  Recheck Cond: ((id = ANY ('{<1000 uuids>}'::uuid[])) AND (namespace_id = '…'::uuid))
  Filter: ((soft_deleted_at IS NULL) AND (npm_package_id = '…'::uuid))
  Rows Removed by Filter: 200
  Heap Blocks: exact=51
  Buffers: shared hit=467
  ->  Bitmap Index Scan on npm_versions_p07_pkey  (cost=0.00..274.35 rows=1000 width=0) (actual time=0.623..0.624 rows=1000 loops=1)
        Index Cond: ((id = ANY ('{<1000 uuids>}'::uuid[])) AND (namespace_id = '…'::uuid))
        Buffers: shared hit=416
Planning:
  Buffers: shared hit=650
Planning Time: 1.456 ms
Execution Time: 0.773 ms

One partition (namespace_id prunes to _p07), and the index scan takes both primary-key columns — id = ANY over the list plus namespace_id — so the id set is resolved entirely from pk_npm_versions. All buffer hits.

The two predicates that are not in that index, npm_package_id and soft_deleted_at IS NULL, are a heap Filter, which is the number worth reading: Rows Removed by Filter: 200. The bitmap heap scan fetches one tuple per requested id, not per resolved one, so a caller that asks about ids which are soft-deleted or owned by a sibling package still pays their heap fetches. It is cheap — a residual filter on already-fetched tuples — and the width=24 projection means none of those fetches detoast package_json, which is the property the doc comment claims. But the cost scales with the size of the id set the caller passes, not with the number of tags that resolve.

Execution is well under planning here (0.77 ms vs 1.46 ms), which is expected with a 1000-element IN list. Figures are stable across five consecutive runs (execution 0.77–0.90 ms; the row and buffer counts identical).

📏 Size

1068 reviewable LOC: 362 datastore (116 production, 246 tests), 675 the pager and its suite, 31 the two supports. Past the 500-line guidance, and this is already the smallest part of the split; the pager and its contract suite are one unit.

Testing

  • Unit, -race, and integration suites green; golangci-lint reports 0 issues, including a --build-tags=integration run.
  • npm conformance (mise run conformance:npm) passes on the head of this stack.
  • The datastore suite covers strictly more than the NpmVersionsByPackageAndVersions table it mirrors, with the cap boundary on the accepting side and cap+1 on the guard side.

Related to #241

Edited by Dzmitry (Dima) Meshcharakou

Merge request reports

Loading
Loading