docs: convert S22 spec and plan into storage-accounting documentation

What this does

S22 is implemented. All 21 step MRs of its plan are merged. The spec and the plan recorded design intent that the code now carries, so this converts them the way commit dddc6cc7b converted S06's:

  • docs/dev/storage-accounting.md grows from the call-site guide it was into the full reference.
  • docs/specs/S22-storage-accounting.md becomes a landing stub, so unanchored inbound links keep working.
  • docs/plans/2026-08-04-s22-storage-accounting.md is deleted.

The new document covers the accounting layers, the counter model and where each delta fires, the namespace_statistics and blob_storage_blobs_by_namespace schema, the emit API and its caller contracts, the buffered drain, the reconciliation pass and its candidate selection, configuration, observability, error cases, security considerations, and the known gaps. The file's existing call-site content is kept rather than restated.

What the conversion corrected

Every claim is checked against main instead of copied from the spec. Where the two disagreed, the document records what the code does:

Spec or plan said main does
Shadow table columns namespace_id, sha256, size Physical order is namespace_id, size, sha256, and the jet model follows it
The size_bytes recompute is one SELECT DISTINCT shape Only the two Maven walks use DISTINCT; the four container and npm walks deduplicate with a UNION of two arms
artifacts_count counts version rows The two container branches count manifest rows
A nine-cell (format, kind) matrix Twelve cells, eight positive hits, six distinct walks — docker and oci share both container walks
components_count sums six per-table counts One statement with six scalar subqueries
Only reconciliation_backlog is lease-gated namespaces_missing_statistics is too; both drop together
A 1000-row cap on drain_batch_size The cap is on reconciliation_batch_size; drain_batch_size defaults to 20000 and has no ceiling
An UPDATE from the shadow's MutableColumns puts size in the WHERE MutableColumns is empty, so it renders with no SET clause and PostgreSQL rejects it
MountBlob mirrors CompleteUpload It emits no namespace-scoped delta, because a mount reuses a blob the namespace already holds
The namespace_statistics migration needs its trigger and seed in one transaction The single file is an observability preference. Only the shadow table's seed needs the atomicity
The composition-root seam is periodicRegistration Renamed to jobRegistration, beside a new workerRegistration

Two decrement sites the design names have no implementation on main, and the document says so in a form that stays true after they land: the lifecycle purger emits no footprint decrements, and nothing reclaims a blob. Both counters come down only at reconciliation until then. Two interim npm writers survive with no caller.

Diff size

Past the 500-line threshold, so here is the split. It is one document conversion, and splitting it leaves either a spec pointing at a document that does not exist or a document duplicating a live spec.

Group Lines
docs/dev/storage-accounting.md +1102 / -22
docs/specs/S22-storage-accounting.md (spec to stub) +24 / -1264
docs/plans/2026-08-04-s22-storage-accounting.md (deleted) -1505
Index, inbound-link, and citation updates, 13 files +38 / -35

Overlap with open merge requests

Four open MRs edit docs/dev/storage-accounting.md, so whichever lands second needs a rebase. Flagging it here because no pipeline reports it:

  • !1895 (merged) rewrites one paragraph this MR turned into a table, and appends a section after text this MR keeps.
  • !1912 (merged) appends a section after the last code block, which this MR also keeps last.
  • !1916 rewrites text inside two sections this MR rewrote.
  • !1997 rewrites the npm first-attach paragraph, which this MR also rewrote.

!1959 edits one row of docs/plans/2026-08-11-s20a-lifecycle-closed-beta.md that this MR also touches. !1512 edits docs/specs/S17-rest-management-api.md around 22 lines above the bullet this MR rewrote. !1953 and !1916 edit docs/specs/S20-a-lifecycle-closed-beta.md, whose one line this MR touches.

The appended sections should apply. The in-place rewrites will conflict.

Testing

Documentation only. No code changes. markdownlint-cli2, lychee --include-fragments, and vale pass on every changed file. Every anchor another document links into the new file resolves, and every file path, symbol, migration name, table, index, trigger, Redis key, job kind, metric, and configuration field the document names exists in the tree.

A second pass re-checked the finished document against the tree and found twelve claims that did not hold. The second commit fixes all twelve and restores five design facts the first pass dropped. Both are listed in that commit's message. The largest were:

  • The management API's emit series carries result="ok"; the three format dispatchers do not.
  • The bulk workers emit inline on a worker goroutine, which the emit-site table's preamble denied.
  • The marshaled payload is 64,660 bytes and 36 over at 862 scopes. The 873/39 pair is the model's arithmetic, which the code says to supersede.
  • The shadow table's seed holds SHARE ROW EXCLUSIVE on blob_storage_blobs and all 64 partitions until it commits, so blob writes are unavailable for its whole duration. That is a deploy-timing constraint, and it was missing.

What review changed

Twelve review threads were worked after the first two commits. Eleven of them moved text, in ten commits; the twelfth was a clean-review note with nothing to change.

  • ## Accounting layers names the three artifact-level write mechanisms separately again, with their call sites. The flattened sentence contradicted S17's Freshness block and left this document's own opening pointing at a section that recorded no mechanism.
  • The tombstone-count paragraph no longer says a lost Δartifacts decrement is lost for good. Every artifacts_count walk carries the soft-delete predicate on each level that has the column, so the recompute corrects it.
  • The reconciliation liveness alert names jobs_processed_total{kind="trigger:reconciliation", status="ok"} and states that a rule written against success matches no series. The foundation-metrics bullet gains ok too.
  • The drain cost model attributes its 80 chunks to both families' ticks, and carries the derivation from the configuration table.
  • The reconciliation guard's Redis-lease rejection is scoped to its critical section, so it no longer reads as a rule about leases beside the collectors that run on one.
  • The error table bounds the no-baseline drop at one reconciliation_orphan_sweep_interval, and gains a row for a key leaving Redis whole.
  • S22's retired criterion numerals are answered: the three cites this MR rewrote name the emit contract instead, and the stub answers every citation elsewhere in one sentence. The management-API emit table regains the one-site invariant the spec's no-retrofit warning carried.
  • docs/specs/S17-rest-management-api.md and docs/plans/2026-08-10-maven-version-size-accounting.md no longer quote text their link targets do not contain.

Two residual items are recorded outside this MR rather than in it. #583 carries the S17 Follow-ups bullet's remaining scope question, and #515 carries the S22 criterion citations in docs/specs/S20-a-lifecycle-closed-beta.md and the two npm emit test files.

Guardrails

  • No specs, plans, ADRs, or e2e catalogs change behavior here, so no conformance run and no catalog revision applies.
  • The three Go comments that name the spec path are left alone: the stub answers them, and editing them would pull three pre-existing over-cap comment blocks into the comment-caps hook.

Related to #515

Edited by Pawel Rozlach

Merge request reports

Loading
Loading