feat(npm): stream the packument render page by page

🎯 What this MR does

Adds RenderPackuments, a streaming packument renderer that walks a package's active version rows through a VersionPager one page at a time and emits the full (kind=0) and abbreviated (kind=2) documents incrementally into caller-supplied io.Writer sinks, dropping each page before it reads the next. Step 1 of 3 in the npm packument streaming generation plan.

No production caller switches here. GeneratePackuments stays in place and untouched; Step 2 moves the cache rebuild onto this renderer and Step 3 converges the cache-miss inline build on the rebuild.

📚 The stack:

Step	MR	What it delivers
1 · this MR	👉 this one	RenderPackuments + VersionPager + golden byte fixtures
2	not yet open	RebuildPackumentCache streams into its blob sessions
3	not yet open	Inline miss converges on the fenced rebuild

🧩 Why streaming

Finding 01 of the AppSec review in work item #241 is an unauthenticated heap-exhaustion DoS: concurrent cache-miss builds across distinct large packages grow live heap without bound in a shared multi-tenant process. !1478 (merged) capped concurrent builds at 64 and said openly that this bounds concurrency, not memory — 64 worst-case builds still reach tens of GB. This MR removes the other factor of the product. Once Steps 2 and 3 land, one build's peak heap is O(page) and the 64-slot cap becomes a real memory bound.

🔍 Two things worth a close read

Emission order is pager order, and the digests move once

versions entries are emitted in the order the pager yields rows, not sorted by version string — sorting needs every row in memory, which is the problem itself. Output stays deterministic for a given row sequence, so the property the content-addressed cache needs (same rows → same bytes → same ETag) holds. The cost is one-time and lands in Step 2: the first rebuild of each package produces a new digest, and a client holding an old ETag revalidates into one extra 200. npm clients treat JSON object key order as meaningless.

Only the versions key is hand-placed

A streamed document cannot let encoding/json sort keys it has not seen yet, so the header fields (_id, _rev, dist-tags, name) are marshaled as one map and the versions key is appended by hand, resting on one stated fact: versions is bytewise-greatest of the top-level keys. This is what remains of the hand-maintained key order the plan names as an accepted smell. TestRenderPackuments_TopLevelKeyOrder pins the emitted order against encoding/json's own sorting of the same key set, so a key emitted out of sorted position fails loudly rather than silently changing every digest.

📏 Diff size

2320 added LOC, over the 500 ceiling in development-model.md. Splitting would not help:

Group	LOC	Why it stays
packument_render.go	506	One renderer; nothing in it stands alone
packument_render_test.go, packument_render_failures_test.go	1127	The contract and failure-path suites for that renderer, including the GeneratePackuments equivalence pin that has no comparator after Step 3
packument_fixtures_test.go	529	The fixture helpers and fakes the suites share; 132 of these lines moved verbatim out of packument_test.go (its -132 in this diff) so Step 3 can delete that file with its subject
packument.go, export_test.go	32	The row helpers take (packageName, baseURL) directly; the test-only buffer-size re-export
Plan, testdata/README.md, 2 goldens	126	The plan puts the golden fixtures in Step 1 on purpose, so the byte-level pin predates the loss of its comparator

The test files are 76% of the diff. Reading packument_render.go plus the golden documents covers the substance.

Spec coverage

Spec: docs/specs/S11-npm-hosted.md, "Packument generation" and "Packument cache". This step re-pins on the streaming path the behavior GeneratePackuments holds today; no spec text changes, and Step 3 owns the spec edits.

Acceptance criteria

#	Criterion	Tests
AC-1	Streamed documents carry the same fields as the materialized generator's	TestRenderPackuments_MatchesGeneratePackuments
AC-2	Exact bytes of both kinds are pinned for a fixture package	TestRenderPackuments_GoldenDocuments
AC-3	Identical rows in identical order yield byte-identical output and distinct per-kind digests; distinct version inputs yield distinct digests	TestRenderPackuments_Deterministic, TestRenderPackuments_DistinctInputsDistinctDigests
AC-4	Top-level key order is encoding/json's sort of the same key set	TestRenderPackuments_TopLevelKeyOrder
AC-5	versions entries follow pager order	TestRenderPackuments_EmitsVersionsInPagerOrder
AC-6	dist-tags is emitted verbatim, keys sorted, {} never null	TestRenderPackuments_EmitsDistTagsVerbatim, TestRenderPackuments_DistTagKeysAreSorted
AC-7	_rev is emitted verbatim in the full document only, omitted when empty	TestRenderPackuments_RevEmittedVerbatimInFullOnly
AC-8	A package with no active versions renders {} for versions and no _rev	TestRenderPackuments_ZeroVersions
AC-9	Abbreviated is a strict projection of full, values agreeing per key	TestRenderPackuments_AbbreviatedIsInstallOnlyProjection
AC-10	dist.tarball is absolute, name percent-encoded, scope stripped from the file name	TestRenderPackuments_TarballURLAbsolute, TestRenderPackuments_DistAbsentEmitsTarballOnly
AC-11	Each page is released before the next is read, and output is emitted as the walk proceeds	TestRenderPackuments_DoesNotRetainPages, TestRenderPackuments_StreamsWhileItWalks
AC-12	Per-row cost stays flat as the version count grows	BenchmarkRenderPackuments
AC-13	Exhaustion is an explicit pager signal; rows arriving alongside done are emitted before the walk ends	TestRenderPackuments_FinalPageMayCarryRows

Error cases

#	Condition	Tests
E-1	Unparseable stored package_json, unparseable dist, invalid version string: fail the render, naming package, version and row id	TestRenderPackuments_FailsClosed
E-2	Pager error mid-walk fails the render	TestRenderPackuments_FailsClosed
E-3	Sink write failure fails the render, either sink	TestRenderPackuments_SinkErrorPropagates
E-4	A sink refused after the other document is terminated still fails the render	TestRenderPackuments_LateSinkFailureLeavesTheOtherComplete
E-5	A nil sink fails before any byte is written	TestRenderPackuments_RejectsNilSink
E-6	Canceled or expired ctx stops the walk at the next row	TestRenderPackuments_ContextDoneAborts

Security considerations

#	Concern	Tests
S-1	BaseURL control bytes must not reach a served dist.tarball; rejected before the first write	TestRenderPackuments_RejectsUnsafeInputs
S-2	A path-traversal package name must fail the row it is projected for	TestRenderPackuments_RejectsUnsafeInputs
S-3	Denylisted and lifecycle-hook fields in a stored package_json must never be re-introduced into a served document	TestRenderPackuments_DropsDeniedAndScriptFields
S-4	A row that cannot be projected must fail the render, never be skipped: a skipped row serves a document missing a version its publisher created	TestRenderPackuments_FailsClosed
S-5	Heap exhaustion (finding 01 of #241)	Partially addressed. This MR bounds the renderer's own live heap to one page; the bound reaches production in Steps 2 and 3, which move the callers onto it.

✅ Verification

• go test ./internal/format/npm/ green, -race included.
• Two guards were re-verified after the buffering change by deleting each and watching its test fail: the nil-sink rejection (PackumentSinks{} panics with a nil pointer dereference without it) and the per-row ctx check (without it, the post-emission cancellation case sees the second row's bytes reach the sink). The golden byte pin held unchanged through every commit here, which is the third guard doing its job.
• golangci-lint run --max-same-issues=0 --max-issues-per-linter=0 internal/format/npm/: 0 issues.
• BenchmarkRenderPackuments shows the flat per-row cost the plan asks for: 82.1 allocs/row and 60,227 B/row at 1,000 versions, against 82.0 and 60,026 at 25,000 (the ADR 004 ceiling).
• Live heap at the largest case, the sample S-5 rests on: across a 25,000-version render (18 KB package_json per row, page size 1,000), HeapAlloc sampled after a forced GC at each of the 25 page boundaries peaked 186 KB above the test's pinned fixture (468.6 MB). The renderer retains no page once emitted; its own live state is the two 64 KiB sink buffers, the staged prefix buffer, and the two streams.
• Both golden documents are byte-identical to what the renderer produced before any review-fix commit, so nothing cached is invalidated by them.

🧪 E2E scenario impact

None: no HTTP handler or job reaches this renderer yet. The npm conformance suite runs in Steps 2 and 3, which are the ones that change protocol-visible bytes or flow — the plan's Testing Strategy says so.

🗒️ Review history

Findings from a branch review are addressed in 954e4512 (nil sink, per-row ctx, the fail-closed doc claim and its missing test case) and e31d630d (the plan's allocation criterion, and the sink-buffering duty Step 2 owes). A second review pass is addressed in 02e1ee0f (a cancellation case that fires after a row is emitted, plus present-tense fixture claims), f6610422 (what a failed render leaves in each sink, the VersionPager exactly-once contract, the nil-pager panic), and cbd24fc4 (the plan's per-session write count and the ADR 004 / npm.max_package_json_size attribution).

A third review pass (13 threads) is addressed in 119dd5be (explicit pager exhaustion and the ordering clause), 05534f8d (row helpers take the name and base URL directly; header fields marshaled as one map), 9b693121 (renderer-owned sink buffering with flush-error propagation, one shared member-separator flag, the nil-pager wording), a447fd1d (the digest cross-check and consistency-fence doc sentences), e47399d8 (shared fixtures file, forBothKinds, presence-before-order pins, distinct-input digests, the golden-update flag rename), aaceb341 and 89e0f861 (plain punctuation; plan sync), and a34e43a0 (the cancellation case sized so a wrongly emitted row is visible through the renderer's buffer).

Related to #241