Unowned buffered counter columns have no reconciliation, so a shed write drifts permanently

Re-scoped on 2026-08-27: the S22 half is delivered

The original issue said that nothing recomputes any buffered counter column. That is no longer true, and the half that changed is the half S22 owned.

internal/accounting/reconcile_task.go, internal/accounting/reconcile_trigger.go, internal/datastore/reconcile_namespace.go and internal/datastore/reconcile_repository.go are on main. Four recompute methods exist, and they cover the four S22 columns:

Column Recompute on main
repositories.artifacts_count RepositoryReconcileStore.RecomputeArtifactsCount
repositories.size_bytes RepositoryReconcileStore.RecomputeSizeBytes
namespace_statistics.components_count NamespaceReconcileStore.RecomputeComponentsCount
namespace_statistics.deduplicated_size_bytes NamespaceReconcileStore.RecomputeDeduplicatedSizeBytes

A shed write on any of those four is repaired on the next pass, bounded by storage_accounting.reconciliation_interval. Work item 515 delivered this.

This issue now covers the columns that reconciliation still does not reach.

What is missing

Every column below is maintained by a fire-and-forget post-commit write, and nothing recomputes any of them. A write that is shed or fails is lost for good.

Column Emit label Owner
npm_packages.versions_count npm_package_publish_counters (internal/format/npm/publish_commit.go:120) none. S11 interim, no replacement spec
npm_packages.tags_count tags_count (internal/format/npm/disttags.go:84) none. Same
repositories.last_updated_at last_updated_at (internal/format/npm/publish_commit.go:117) none. Work item 442
npm_packages.last_downloaded_at, npm_versions.last_downloaded_at last_downloaded_at (internal/format/npm/download.go:376) S18
npm remote last_downloaded_at npm_remote_last_downloaded_at (internal/format/npm/remote_tarball.go:85) S18
repositories.downloads_count downloads_count (internal/format/maven/buffered.go:61) S18
Maven last_downloaded_at last_downloaded_at (internal/format/maven/buffered.go:57) S18
maven_versions.size_bytes version_size_bytes (internal/format/maven/buffered.go:66) work item 550

The shedding itself is specified. S11's "Interim implementation of buffered columns" says a failed emission must not surface to the client, and the counters are lossy by design. The gap is the other half. ADR-011 covers data reconciliation, and none of the columns above has a pass.

Why it matters

Two of these are enforcement inputs, not display values.

npm_packages.versions_count backs ADR-004's 25,000-version cap. A lost increment under-counts, so the cap admits more than it should.

npm_packages.tags_count has the same shape and no cap behind it today.

A dropped write is invisible except as a metric label, so drift accumulates with no signal on the affected row.

The ownership gap is what blocks this

The columns above have three owners and one hole.

  • S18 owns the last_downloaded_at timestamps and downloads_count.
  • Work item 550 owns maven_versions.size_bytes.
  • Work item 442 owns repositories.last_updated_at, which has no mechanism and no spec.
  • npm_packages.versions_count and tags_count stay on npm's own mechanism with no replacement spec (S11).

Reconciliation for the unowned columns therefore has nobody to land it. That is the decision this issue needs before it needs an implementation.

Possible shapes

The four S22 columns took shape 1. The reference implementation now exists, which was not true when this issue was filed.

  1. Reconciliation pass per column — periodic recompute from source rows. Repairs drift whatever caused it. internal/accounting/reconcile_task.go is the working example, and River's periodic scheduling is available.
  2. Durable intent — enqueue the delta in the writing transaction through jobsriver.EnqueueTx (internal/jobsriver/enqueue.go:139) and apply it in a worker. Removes the loss instead of repairing it, and stays an asynchronous write per ADR-007.

Both are compatible. Shape 1 is the safety net, and shape 2 removes most of the need for it.

Related

  • #919 (closed) — reconciliation exists now, and its hand-written version-type table list has no exhaustiveness guard. That is the failure mode where the repair is itself wrong.
  • #442 — owns repositories.last_updated_at, one row of the table above.
  • #550 (closed) — owns maven_versions.size_bytes, another row.
  • #559 — derives the in-flight caps from the pool's capacity. Connection-pool starvation, not lost data.
  • Work item 91 centralizes the helpers into a shared package. Code organization, not durability.

Found while reviewing !1403 (merged), which adds two bufferedUpdate emit sites for the unpublish tags_count reset. A shed write there leaves the pre-unpublish count on a tombstoned package.

Edited by Pawel Rozlach