## Summary An npm packument blob's bytes reach no storage counter, on any path. `npm_metadata_files` rows hold one packument blob per kind, at most three per package. `recomputeNpmFilesSizeStmt` unions those digests into the npm branch of the repository size walk, and the by-namespace shadow trigger fires on their `blob_storage_blobs` rows. Both recomputes therefore count packument bytes, and no emit site anywhere carries them. Three defects, one root. Two are the write half and one is the delete half. This issue was split out of https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/762, which bundled them with an ADR-007 amendment obligation that no merge request in this repository can discharge. It absorbs https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/761, which held the delete half on its own. ## The write half: no emit site exists on the rebuild path Copied unchanged from work item 762, where these were gaps 2 and 3. 2. **The repository-scoped `Δsize` never carries packument bytes.** [ADR-007's Repository-level storage accounting reconciliation](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/artifact_registry/decisions/007_database_schema/#repository-level-storage-accounting-reconciliation) defines the npm repository footprint as `npm_files` in union with `npm_metadata_files`. `recomputeNpmFilesSizeStmt` implements both arms. No emit site covers the packument blob. `UpsertNpmMetadataFileForBlob` writes that row from the packument cache rebuild, and no emit site sits on that path. 3. **The namespace-scoped `Δdedup_size` never carries packument bytes either.** `RecomputeDeduplicatedSizeBytes` sums the `blob_storage_blobs_by_namespace` shadow table. A trigger populates that table for every blob row, packument blobs included. ### Why the namespace gap matters most The S22 spec names `deduplicated_size_bytes` the byte-based billing input. This counter under-reports, and it under-reports in the customer's favor. Nobody reports an error of that direction, so the gap can persist without a report from anyone. ### The shape of the drift `UpsertNpmMetadataFileForBlob` creates the new attachment and removes the previous one in one transaction. A package therefore holds one packument blob per kind at a time, and the blob is replaced rather than accumulated. Each rebuild is both a first attach of a new digest and a last detach of the old one, and the fast path sees neither. The drift is not unbounded growth of one packument per publish. After a reconciliation pass writes the true value, the fast path moves away from it again on each later rebuild, by the size difference between the two packuments. ## The delete half: two routes hard-delete the rows and carry none of their bytes Copied unchanged from work item 761. Two npm delete routes both tombstone and hard-delete inside one transaction, and only the tombstone half reaches `repositories.size_bytes`. This issue owns both arms: - The management package `DELETE`, which hard-deletes at `internal/datastore/npm_package_management_deleter.go:271`. - The protocol whole-package unpublish, which hard-deletes at `internal/datastore/npm_package_unpublish_deleter.go:254`. Both routes make the same call, to `NpmMetadataFileStore.DeleteNpmMetadataFiles`, and one merge request can close both arms through it. `NpmPackageManagementDeleter.cascade` marks the package, its versions, and its files, then calls `removeCacheRows`, which hard-deletes the package's `npm_metadata_files` rows and the `blob_storage_attachments` rows they held, inside the same transaction. The handler arm emits `deleteCounterTargetTombstone`, whose delta set is `Δartifacts` alone, because the footprint counters settle when the lifecycle purger reaps the marked rows. `NpmPackageUnpublishDeleter.cascade` has the same shape. It marks the package with `SoftDeleteNpmPackage` and `CascadeSoftDeleteNpmPackage`, then hard-deletes the same rows with `DeleteNpmMetadataFiles`. Its handler emits `Δsize = 0` at `internal/format/npm/unpublish_package.go:263`, on the same reasoning about the purger. That reasoning holds for the marked rows and not for the hard-deleted ones. The mark is not what removes the packument's bytes from the size walk; the hard delete is. The purger can never settle them either, because the rows are already gone by the time it reaches the tombstone. `repositories.size_bytes` therefore reads high by the packument blobs' size from the moment either delete commits until a reconciliation pass recomputes that scope. ### Why this is not a delete-emit defect S22 criterion 37's delta table has no row for a target that both tombstones and hard-deletes, so the gap sits in the criterion's model rather than in the emit that implements it. `deleteCounterTarget` cannot express the needed delta set, a tombstone that also frees bytes, so closing it is a widening rather than a one-line change. ## What closing this needs **The write half needs an emit site and a datastore change.** `UpsertNpmMetadataFileForBlob` returns only `error`, and it never reports the old digest that it removes. The namespace half has a cheaper signal: `packument_cache.go` discards a `storage.StoreResult` whose `Deduplicated` field is the namespace first-attach predicate. **The delete half needs a fourth `deleteCounterTarget` value, or a delta set carrying both amounts.** `NpmMetadataFileStore.DeleteNpmMetadataFiles` already runs `DELETE ... RETURNING` and reports each removed row's attachment coordinates. It can report the bytes freed on the repository's last reference through the `repoFreedBlobSizeTx` helper, and the handler arm can carry that into the emit beside `hiddenArtifacts`. Both delete arms reach that one helper, so one merge request can close both. The unpublish arm needs two more changes of its own: `NpmPackageUnpublishDeleter.cascade` takes no repository id today, and `UnpublishPackage` must carry the freed bytes back to its handler. An amendment to S22 criterion 37's npm-package row belongs with the delete half, so the spec's table and the emit agree on what the arm owes. ## Depends on !1997, and the dependency is the repository predicate The repository-scoped half of the write gap needs to know whether the repository already references a digest. !1997 introduces `NpmFileStore.RepositoryHoldsBlob` for exactly that question, on the npm publish path, and that method is on its branch rather than on `main`. **Merge order.** Build the write half's repository arm on top of !1997 rather than beside it. Writing a second membership probe while that one is open produces two answers to one question, in one package. https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/929 reports that `RepositoryHoldsBlob` fans out namespace-wide on the tarball digest. Whatever that issue settles about the query shape applies here too, because this issue's repository arm is its second consumer. ## One caution for the implementer [ADR-007's Cleanup tasks](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/artifact_registry/decisions/007_database_schema/#cleanup-tasks) puts attachment removal in the application's transaction, and ADR-007's repository accounting sentence says the counter decrements when garbage collection hard-deletes the last attachment. For a packument the attachment goes inside `UpsertNpmMetadataFileForBlob`, which is the application's own transaction. A literal reading of that ADR-007 clause therefore never decrements, and the column climbs without bound. The S22 spec carries the workable rule, and it is what to follow here. Work item 762 owns the amendment that reconciles the ADR text with the tree. This issue does not wait on it: the amendment moves the ADR's decrement clause for Container/OCI, and nothing here needs that clause to move first. ## Scale and bound At most three `npm_metadata_files` rows per package, each holding one packument blob. `RepositoryReconcileStore.WriteBackCounters` takes an absolute `sizeBytes` and overrides the column, so no pass inherits a wrong value. The pass also selects namespaces by staleness rather than from the dirty set, so it reaches the repository whether or not an emit marked it. `reconciliation_interval` is both the tick period and the staleness threshold, and `config.example.yaml` defaults it to `1h`. A namespace reconciled at `T` is not a stale candidate before `T + reconciliation_interval`, and the next tick that re-selects it can be one interval later again. A wrong value therefore survives about one to two intervals. ## Related - https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/762 — the two ADR-007 handbook amendments this issue was split from. - https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/910 — the other npm counter defect the split left standing. It needs a namespace attachment probe rather than the repository one, so it is a sibling rather than part of this. - https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/929 — the query shape of the probe this issue's repository arm consumes. Related to https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/515