npm publish: a retried publish emits no namespace dedup bytes when an earlier attempt committed the blob

Summary

When an npm publish commits its CAS blob and then fails before its row commit, the blob row survives in the namespace shadow table and no counter emit happens for it. When that publish is retried, the CAS commit reports the blob as deduplicated, so the retry emits Δdedup_size = 0. From that point namespace_statistics.deduplicated_size_bytes is lower than what a reconciliation pass recomputes, and it stays lower until a pass runs.

The gap is permanent for the fast path and invisible to it. Only reconciliation closes it.

Why it happens

The CAS commit and the row commit are separate operations, and the emit is attached to the second one.

PublishHandler receives blobDeduplicated as the CAS commit's storage.StoreResult.Deduplicated verdict, so the blob is already stored by the time the handler runs (internal/format/npm/publish_commit.go:127-135). The row commit runs after that, at CommitPublish (:203). The emit runs after the row commit, through afterCommit into emitPublishCounters (:227, :300).

So an error between the CAS commit and a successful CommitPublish leaves the blob stored with no emit for it. The blob is real, it is reachable from the namespace shadow, and a reconciliation pass counts it.

On the retry, the CAS commit finds the digest present and reports Deduplicated = true. emitPublishCounters reads that verdict as "the namespace already held these bytes" and sends Δdedup_size = 0 (:320-352). That reading is correct about the storage and wrong about the counter: the bytes were never counted, because the attempt that stored them never emitted.

When it is observable

The condition is a publish whose CAS commit succeeded and whose row commit did not. ErrNpmVersionExists is not this case — that is a rejected duplicate version, which never stored a new digest. The cases that qualify are an error or a crash between the two commits.

Once such an attempt has happened for a digest, every later publish of those same bytes into that namespace emits zero for the namespace counter, because the digest is deduplicated from then on.

The divergence is bounded by the reconciliation interval where a pass runs, and unbounded where one does not.

What closes it

Options, in the order they occurred to the reporter rather than in order of preference:

1. Carry the emit off the row commit and onto the CAS commit, so the bytes are counted when they are stored. This changes what the counter means during a failed publish: the namespace is charged for bytes no artifact references yet.
2. Have the retry path distinguish "deduplicated against a referenced blob" from "deduplicated against an unreferenced one", and emit the bytes in the second case. This needs a reference probe the publish path does not have today.
3. Leave the fast path alone and rely on reconciliation, recording the divergence as accepted. This is what happens now, undocumented.

Option 3 is what the code does today. Whether that is acceptable depends on how often the window is hit, and nothing measures it.

Related

• #497 is the same shape on a different path: a refused packument rebuild that has already committed its blob. Its harm is a leaked unreferenced blob; this issue's harm is a counter that under-counts. Different producer, different consequence, same root — a committed CAS blob whose owning operation did not complete.
• #498 covers the absence of a reclaimer for unreferenced committed blobs, and names itself as the systemic problem behind individual producers.
• #762 fixes three npm counter gaps and this is not one of them. It was previously mentioned in a CounterEmitter doc paragraph that attributed it to #762; when that paragraph was rewritten the pointer had nowhere to go, which is why this issue exists.

Labels mirror #497, the closest sibling, except that bug::functional replaces bug::performance: the defect is a wrong counter value rather than a cost.

This is a bot message 🤖 — /smurfit