feat(npm): book a delivered tarball on the download count part 3/3 (repository-column-writers plan: 4/5)

Part 3 of 3 of the repository download-count family, and the part that makes it live. Adds the leader-elected drain trigger, binds every kind to a worker, and gives the family its first producer.

What this adds

  • DownloadsDrainTriggerWorker and its schedule, plus the registration that binds part 2's chunk worker and this trigger to their kinds. Without it the family would be a write-only sink.
  • npm's producer: a delivered hosted tarball GET books +1. Delivery is the gate the retention bump and artifact_pulled already share, so a HEAD, a 304, a 404 and a transfer that ended short of the declared length each book nothing. The increment is dispatched off the request path.
  • The counter-emit dispatcher now takes the column its site moves, so a shed or panicked download emit books under column="downloads_count" rather than landing on the byte series. Maven books that same label value for a buffered UPDATE that reports ok, so only dispatch volume compares across the two formats; the constants and both metric rows say so.
  • Each of the dispatcher's six emit sites now holds its own in-flight quota, summing to the existing cap of 64 rather than raising it. The delivered tarball GET is both the highest-volume site and the only one whose shed is permanent, so a flat cap let a Redis slowdown spend the pool on the deltas that can never be repaired while starving the five that heal. The site slot is taken before the global one, so the global shed arm is now unreachable and every shed books against the site's own reserve. internal/format/npm/counter_emit_quotas.md carries the number and the reason per site; arrival rate sizes none of them, because no signal in the tree reports a per-site dispatch rate.

Merge order

This part is what closes the window parts 1 and 2 leave open. Parts 1 and 2 merge with the family unregistered and unproduced; this part adds the trigger, the registration and the producer together rather than widening one ahead of the others. It does not build on its own.

What this does not reach

  • kind=remote npm tarball reads book no download count, so a remote repository's column stays at zero however much is pulled through it. Only the hosted route has a producer.
  • Container/OCI and Maven download counting are unchanged. Maven still books through its own per-read writer.
  • No reconciliation pass recomputes repositories.downloads_count, so every shed dispatch, every lost Redis reply and every drain-versus-drain over-count on this family is permanent rather than bounded. The ## Error cases table in docs/dev/storage-accounting.md states the outcome per family for every row, and the rollout window is documented in docs/dev/background-jobs.md: an old pod claims the new chunk kind, fails it inside River rather than inside runChunk, and no recovery arm re-adds the scopes the tick already popped. docs/dev/alerting.md gains the required alert that follows from this, since reconciliation's three do not cover a family it never recomputes: any increase in npm_buffered_counter_updates_total{column="downloads_count", result="dropped"} rather than a failure to trend toward zero. The rollout-window loss sheds no dispatch, so that series does not move for it and no series names it; the entry says so rather than implying coverage it does not have.

Stack

Merge in this order. Each part builds and tests on its own, and the family is inert until part 3. Each part is hard-gated behind its parent, so none can merge early.

# MR What it adds Reviewable LOC
1 !2494 (merged) The SQL half: the delta type, the two statements, the single-row writer 1457
2 !2495 (merged) The Redis half, Emitter.EmitRepoDownloads, the chunk worker 1430
3 !2496 (merged) The drain trigger, the registration, npm's producer, every doc surface 1665

Diff size

Guardrail 18 asks for a split or a justification past 500 reviewable LOC. This step was split three ways and each part is still over it, so the justification stands per part rather than being cleared by the split.

By file group: Go tests 13 files +594/-143 = 737, Go production 18 files +219/-223 = 442, docs/ 8 files +208/-90 = 298, sidecar .md and the run recipe 4 files +136/-37 = 173, yaml 2 files +10/-5 = 15. Just under three quarters is documentation and tests. The production half cannot be split further without reopening the window above: the trigger, the registration and the producer have to land together.

Testing

Full unit suites, plus internal/accounting, internal/datastore and cmd/artifact-registry integration suites. Adds the composition-root case for the download seam, which the publish, rebuild and remote-fill seams already had and this one did not.

Guardrail 12: no wire-protocol behaviour changes on any format, so the conformance suites do not apply. Guardrail 13: docs/testing/e2e/npm.md gains e2e.npm.consume.repository-downloads-count and its usage-data row. Guardrail 21: the run recipe gains the third family's walkthrough.

Related to #1223 (closed)

Edited by Dzmitry (Dima) Meshcharakou

Merge request reports

Loading
Loading