Coalesce write-path packument rebuilds per package instead of retiring the singleflight key

Summary

Coalesce write-path packument rebuild dispatches per package, replacing enqueueRebuildAfterWrite's singleflight.Forget + fresh-leader approach with a "rebuild again when the current one finishes" flag.

This is the more correct answer to the concurrency regression !1262 (merged) introduced. That MR shipped a per-package sub-quota instead (rebuildMaxPerPackage), which bounds the damage but does not remove it.

Background

enqueueRebuildAfterWrite retires the singleflight key so a write-triggered rebuild can never collapse into one that read npm_versions before the write committed (S11 AC 27). The cost is that per-package rebuild concurrency goes from 1 to the in-flight cap.

Reviewed on !1262 (merged), the accepted cost was larger than the MR's note stated:

  • rebuildTimeout is 30s (internal/format/npm/packument_cache.go:286) and each rebuild does a full keyset scan of npm_versions + npm_tags plus three CAS blob writes to object storage, so the window is wide.
  • PUT/DELETE .../dist-tags/{tag} are the cheapest possible writes with no rate limiting at this layer, so driving it is a shell loop.
  • The slots now hold concurrently rendering leaders rather than idle followers. internal/format/npm/inline_build.go:29 documents the same render at up to ~25,000 versions x ~20 KB.
  • rebuildSem is process-wide and sheds on saturation, so one hot package starves every other package's rebuilds into result=dropped, and every read then pays a full inline build — compounding the load.

What !1262 (merged) shipped instead

A per-package sub-quota capping how many of the in-flight slots one package can hold. That bounds cross-package starvation, which is the regression the MR actually introduced. It does not restore per-package concurrency to 1, and it does not address the point below.

Why coalescing is better

A flag ("a rebuild is running; run one more when it finishes") keeps per-package concurrency at 1 and guarantees the write gets a post-commit rebuild. The current design guarantees the second only by spawning a new leader, which is what raises concurrency.

It also closes a gap the sub-quota leaves open: a shed write-path dispatch means that write gets no rebuild at all, with nothing retrying. TestEnqueueRebuildAfterWrite_ShedsWhenSaturated currently pins that as the intended contract. That gap predates !1262 (merged) and was deliberately left alone there, but coalescing is the change that would let it be fixed — the pending flag survives a shed.

Why it was deferred

It replaces the retire mechanism !1262 (merged) is built on, and that MR was already large and carrying a schema migration. Doing it there would have meant redesigning the mechanism in the same MR that introduced it.

Note on the eventual jobsriver swap

enqueueRebuildAfterWrite already carries a TODO(s27-cache-rebuild) warning that River's uniqueness must not dedup a write-path enqueue against a RUNNING job, because that reinstates exactly the collapse the retire exists to prevent. Coalescing is the same idea expressed deliberately, so this issue and that TODO should be resolved together — a pending-flag design maps cleanly onto "dedup against PENDING, never against RUNNING".

Related to !1262 (merged).