feat(npm): book a delivered tarball on the download count part 3/3 (repository-column-writers plan: 4/5)
Part 3 of 3 of the repository download-count family, and the part that makes it live. Adds the leader-elected drain trigger, binds every kind to a worker, and gives the family its first producer.
What this adds
DownloadsDrainTriggerWorkerand its schedule, plus the registration that binds part 2's chunk worker and this trigger to their kinds. Without it the family would be a write-only sink.- npm's producer: a delivered hosted tarball
GETbooks+1. Delivery is the gate the retention bump andartifact_pulledalready share, so aHEAD, a304, a404and a transfer that ended short of the declared length each book nothing. The increment is dispatched off the request path. - The counter-emit dispatcher now takes the column its site moves, so a shed
or panicked download emit books under
column="downloads_count"rather than landing on the byte series. Maven books that same label value for a bufferedUPDATEthat reportsok, so only dispatch volume compares across the two formats; the constants and both metric rows say so. - Each of the dispatcher's six emit sites now holds its own in-flight quota,
summing to the existing cap of 64 rather than raising it. The delivered
tarball
GETis both the highest-volume site and the only one whose shed is permanent, so a flat cap let a Redis slowdown spend the pool on the deltas that can never be repaired while starving the five that heal. The site slot is taken before the global one, so the global shed arm is now unreachable and every shed books against the site's own reserve.internal/format/npm/counter_emit_quotas.mdcarries the number and the reason per site; arrival rate sizes none of them, because no signal in the tree reports a per-site dispatch rate.
Merge order
This part is what closes the window parts 1 and 2 leave open. Parts 1 and 2 merge with the family unregistered and unproduced; this part adds the trigger, the registration and the producer together rather than widening one ahead of the others. It does not build on its own.
What this does not reach
kind=remotenpm tarball reads book no download count, so a remote repository's column stays at zero however much is pulled through it. Only the hosted route has a producer.- Container/OCI and Maven download counting are unchanged. Maven still books through its own per-read writer.
- No reconciliation pass recomputes
repositories.downloads_count, so every shed dispatch, every lost Redis reply and every drain-versus-drain over-count on this family is permanent rather than bounded. The## Error casestable indocs/dev/storage-accounting.mdstates the outcome per family for every row, and the rollout window is documented indocs/dev/background-jobs.md: an old pod claims the new chunk kind, fails it inside River rather than insiderunChunk, and no recovery arm re-adds the scopes the tick already popped.docs/dev/alerting.mdgains the required alert that follows from this, since reconciliation's three do not cover a family it never recomputes: any increase innpm_buffered_counter_updates_total{column="downloads_count", result="dropped"}rather than a failure to trend toward zero. The rollout-window loss sheds no dispatch, so that series does not move for it and no series names it; the entry says so rather than implying coverage it does not have.
Stack
Merge in this order. Each part builds and tests on its own, and the family is inert until part 3. Each part is hard-gated behind its parent, so none can merge early.
| # | MR | What it adds | Reviewable LOC |
|---|---|---|---|
| 1 | !2494 (merged) | The SQL half: the delta type, the two statements, the single-row writer | 1457 |
| 2 | !2495 (merged) | The Redis half, Emitter.EmitRepoDownloads, the chunk worker |
1430 |
| 3 | !2496 (merged) | The drain trigger, the registration, npm's producer, every doc surface | 1665 |
Diff size
Guardrail 18 asks for a split or a justification past 500 reviewable LOC. This step was split three ways and each part is still over it, so the justification stands per part rather than being cleared by the split.
By file group: Go tests 13 files +594/-143 = 737, Go production 18 files
+219/-223 = 442, docs/ 8 files +208/-90 = 298, sidecar .md and the run
recipe 4 files +136/-37 = 173, yaml 2 files +10/-5 = 15. Just under three
quarters is documentation and tests. The production half cannot be split
further without reopening the window above: the trigger, the registration
and the producer have to land together.
Testing
Full unit suites, plus internal/accounting, internal/datastore and
cmd/artifact-registry integration suites. Adds the composition-root case
for the download seam, which the publish, rebuild and remote-fill seams
already had and this one did not.
Guardrail 12: no wire-protocol behaviour changes on any format, so the
conformance suites do not apply. Guardrail 13:
docs/testing/e2e/npm.md gains e2e.npm.consume.repository-downloads-count
and its usage-data row. Guardrail 21: the run recipe gains the third family's
walkthrough.
Related to #1223 (closed)