feat(accounting): leased tombstone-count gauge and the count behind it (S20-A plan: 17b/21)

What this step delivers

This step adds gitlab_artifact_registry_lifecycle_tombstones, a gauge that carries the number of rows that hold a soft_deleted_at tombstone across every namespace. The one label is purge_scope. Its two values name the purge level that reaps the row: repository and artifact.

The gauge registers through internal/accounting as a leased scrape-time collector. It holds the same Redis mutual-exclusion lease as the reconciliation-backlog collector, so at most one set of series exists fleet-wide. A pod that holds no lease exposes no series at all. The holder serves the gauge from a cache and re-runs the count behind it at most once per lifecycle.sweep_interval, which ## The count is floored at the sweep interval explains.

The registration sits in cmd/artifact-registry/wire_accounting_metrics.go, beside the backlog collector, and not in accounting.RegisterMetrics. A scrape-time collector needs a store over the database pool and a started lease, and the RegisterMetrics call site holds neither.

The sample behind the gauge is LifecycleScanStore.CountTombstonesByLevel. It runs eight COUNT(*) statements over the tombstone-discovery indexes. It returns both levels every time. When a level holds no tombstone, the value is zero. It applies no retention window, so a row counts from the instant its tombstone is written. The collector caches what it returns, so a scrape inside the sweep interval re-emits that number and issues no statement.

docs/dev/observability.md gains the catalog row for the new family. .claude/skills/run-artifact-registry/SKILL.md now names the third lease-gated gauge family, and a re-run of driver.sh smoke gave 48 passed and 0 failed.

Spec coverage

Spec: docs/specs/S20-a-lifecycle-closed-beta.md

Acceptance criteria

# Criterion Tests
AC-1 Repository tombstone discovery honours the retention window Not this step. Covered by TestLifecycleScanStore_ScanTombstonedRepositories.
AC-2 A live row is never a purge root Partly this step, for the count rather than the scan: TestLifecycleScanStore_CountTombstonesByLevel/a_live_repository_is_not_counted..., .../a_live_artifact_row_is_not_counted...
AC-3 Three package/image scans are index-backed, one partition Not this step. Purger discovery scope.
AC-4 Five version-level scans are index-backed, one partition Not this step. Purger discovery scope.
AC-5 Re-running a purge on a reaped subtree is a no-op Not this step. Purge worker scope.
AC-6 The npm repository walk is state-blind Not this step. Reap scope.
AC-7 Every purge transaction is bounded Not this step. Chunk-driver scope.
AC-8 A frozen namespace's purge refuses retryably Not this step. Purge worker scope.
AC-9 npm version reap Not this step. Reap scope.
AC-10 Unpublish-then-republish leaves no orphan Not this step. Reap scope.
AC-11 npm package reap Not this step. Reap scope.
AC-12 DeleteNpmMetadataFiles removes attachments in-transaction Not this step. Reap scope.
AC-13 Maven version and package reap Not this step. Reap scope.
AC-14 Container repository reap through the per-artifact deleters Not this step. Reap scope.
AC-15 Index-and-children reaped parent-first in one purge Not this step. Reap scope.
AC-16 Tombstoned image under a live repository is reaped Not this step. Reap scope.
AC-17 The container repository walk is state-blind Not this step. Reap scope.
AC-18 Remote-cache subtree reap Not this step. Reap scope.
AC-19 container_remote_manifests and container_remote_blobs reap Not this step. Reap scope.
AC-20 Zero attachments remain after a reap Not this step. Reap scope.
AC-21 DELETE without destructive returns 400 Not this step. Repository entry point.
AC-22 destructive=false on an empty repository returns 204 Not this step. Repository entry point.
AC-23 destructive=false on a non-empty repository returns 409 Not this step. Repository entry point.
AC-24 That 409's message names the caller's remedy Not this step. Repository entry point.
AC-25 destructive=true returns 202 and enqueues transactionally Not this step. Repository entry point.
AC-26 destructive=true on an empty repository returns 204 Not this step. Repository entry point.
AC-27 Every client route 404s after the 202 Not this step. Repository entry point.
AC-28 Create at a tombstoned name returns 409 until the finalizer runs Not this step. Repository entry point.
AC-29 The finalizer cannot succeed early Not this step. Purge worker scope.
AC-30 A finalizer over an empty subtree removes the row and its links Not this step. Purge worker scope.
AC-31 A tombstone with no live job is re-queued within one interval Not this step. Sweep scope.
AC-32 The sweep reads oldest-first Not this step. Sweep scope.
AC-33 A version-less tombstoned npm_packages row is enumerated Not this step. Sweep scope. Its count half: TestLifecycleScanStore_CountTombstonesByLevel/a_tombstoned_npm_packages_row_moves_the_artifact_level_and_no_other
AC-34 A tombstone with a live job is not re-queued Not this step. Sweep scope.
AC-35 A terminal prior job is re-queued next tick Not this step. Sweep scope.
AC-36 Narrowed. A permanently failing purge keeps its repository counted in the tombstone-count gauge, and re-queued at the bounded once-per-interval cadence Gauge half only: TestTombstoneCollector_APermanentlyStalledPurgeStaysCounted. The re-queue half is the sweep's and is left to the step that adds it; the failing-purge framing needs the purge worker, which is on neither this branch nor main.
AC-37 The sweep registers a worker for its periodic kind Not this step. Sweep scope.
AC-38 A frozen namespace is skipped and re-enumerated when it clears Not this step. Sweep scope. The gauge keeps counting a frozen namespace's rows by construction: the count applies no serviceability predicate.
AC-39 FindByNameInNamespace hides a tombstoned repository Not this step. Tombstone-visibility scope.
AC-40 RepositoryStore.FindByID hides a tombstoned repository Not this step. Tombstone-visibility scope.
AC-41 RepositoryStore.Update hides a tombstoned repository Not this step. Tombstone-visibility scope.
AC-42 Five statements still act on a tombstoned row Not this step. Tombstone-visibility scope.
AC-43 parentRepositoryIsActive refuses a tombstoned parent Not this step. Tombstone-visibility scope.
AC-44 Four container_images gates hide a tombstoned image Not this step. Tombstone-visibility scope.
AC-45 /v2/ routes 404 at the image tier Not this step. Tombstone-visibility scope.
AC-46 Blob mount source answers 202 Not this step. Tombstone-visibility scope.
AC-47 A tombstoned image named in from= Not this step. Tombstone-visibility scope.
AC-48 The mount destination is not a 404 case Not this step. Tombstone-visibility scope.
AC-49 UpsertContainerImage does not resurrect a tombstoned image Not this step. Tombstone-visibility scope.
AC-50 ContainerImageStore.Delete still acts on a tombstoned row Not this step. Tombstone-visibility scope.
AC-51 The tombstone write emits no counter delta Not this step. Accounting call-site scope.
AC-52 Purge emits the level's delta set Not this step. Accounting call-site scope.
AC-53 Every emit site commits its source rows first Not this step. Accounting call-site scope.
AC-54 A chunk emits from its committed affected-row aggregate Not this step. Accounting call-site scope.
AC-55 No raw foreign-key cascade is used Not this step. Accounting call-site scope.
AC-56 Three index migrations are in structure.sql Not this step. Schema scope.
AC-57 No column or CHECK is added Not this step. Schema scope.
AC-58 unique_container_images_ns_id_cr_id_name stays partial Not this step. Schema scope.
AC-59 Config load accepts and rejects the documented windows Not this step. Configuration scope.

Step 17b also carries three acceptance statements the plan states rather than the spec.

Plan acceptance Tests
The tombstone-count gauge is labeled by level TestTombstoneCollector_ExposesOneGaugeLabeledByPurgeScope, TestTombstoneCollector_LeaseHolderReportsBothLevels, TestTombstoneCollector_ReportsZeroForALevelHoldingNoTombstone, TestTombstoneCollector_ALevelOutsideTheDefinedSetEmitsNoSeries, TestTombstoneCollector_ALevelTheStoreOmitsGetsNoInventedZero
A pod not holding the lease emits no sample, asserted directly TestTombstoneCollector_TwoInstancesExactlyOneEmits, TestTombstoneCollector_LosingTheLeaseStopsTheSamples
Label cardinality passes the internal/metrics audit, and no label carries a namespace or repository identifier TestTombstoneCollector_PassesTheCardinalityAudit, TestTombstoneCollector_ExposesOneGaugeLabeledByPurgeScope

The sample's own contract, which the plan names but gives no acceptance text for.

Contract Tests
Call guards reject before any statement runs TestLifecycleScanStore_CountTombstonesByLevel_Guards
Both levels are always reported, zero-valued when empty TestLifecycleScanStore_CountTombstonesByLevel/both_levels_are_reported_even_when_nothing_new_was_tombstoned, TestTombstoneCollector_ReportsZeroForALevelHoldingNoTombstone
Each of the seven artifact tables is a positive hit TestLifecycleScanStore_CountTombstonesByLevel/a_tombstoned_<table>_row_moves_the_artifact_level_and_no_other, one subtest per table
The count is fleet-wide, not namespace-scoped TestLifecycleScanStore_CountTombstonesByLevel/the_count_spans_every_namespace
No retention-window filter TestLifecycleScanStore_CountTombstonesByLevel/a_tombstone_written_a_moment_ago_is_counted,_whatever_retention_window_is_configured
A failed count is reported, not papered over TestLifecycleScanStore_CountTombstonesByLevel/query_failure_wraps_the_cause_and_names_the_operation, in internal/datastore/lifecycle_scan_integration_test.go, and TestTombstoneCollector_AFailedCountBeforeTheFirstSucceedsEmitsNothing, TestTombstoneCollector_AFailedCountReEmitsTheCachedNumber, TestTombstoneCollector_AFailedCountIsNotRetriedInsideTheInterval
The count is bounded, and floored at the sweep interval rather than taken per scrape TestTombstoneCollector_TheCountCarriesADeadline, TestTombstoneCollector_ScrapesInsideTheIntervalShareOneCount, TestTombstoneCollector_RecountsPastTheInterval, TestTombstoneCollector_ConcurrentScrapesIssueOneCount
Construction refuses a nil dependency or a non-positive interval TestNewTombstoneCollector_PanicsOnANilDependency, TestNewTombstoneCollector_PanicsOnANonPositiveSweepInterval, TestNewTombstoneCollector_AcceptsItsDependencies

Error cases

# Condition Tests
E-1 DELETE omits destructive Not this step. Repository entry point.
E-2 destructive carries a non-boolean value Not this step. Repository entry point.
E-3 destructive=false on a non-empty repository Not this step. Repository entry point.
E-4 A publish races the DELETE FROM repositories Not this step. Repository entry point.
E-5 DELETE on a missing or already-tombstoned repository Not this step. Repository entry point.
E-6 Create at a name held by a tombstoned repository Not this step. Repository entry point.
E-7 Any client route naming a tombstoned repository Not this step. Tombstone-visibility scope.
E-8 Blob mount whose from= names a tombstoned repository Not this step. Tombstone-visibility scope.
E-9 Purge job: 23503 on a parent delete. The sweep re-queues it and the tombstone-count gauge keeps counting it Gauge half: TestTombstoneCollector_APermanentlyStalledPurgeStaysCounted. The 23503 classification and the re-queue belong to the reap and sweep steps.
E-10 ContainerManifestDeleter refuses a still-referenced manifest Not this step. Reap scope.
E-11 The finalizer aborts because an artifact remains Not this step. Purge worker scope.
E-12 Repository already hard-deleted when the job runs Not this step. Purge worker scope.
E-13 Retention window changed between enqueue and run Not this step. Purge worker scope. The gauge is unaffected by construction: it applies no window.
E-14 Sweep: enumeration query fails Not this step. Sweep scope.
E-15 Sweep: interval elapses before enumeration completes Not this step. Sweep scope.
E-16 Emit API unavailable at a purge or tombstone site Not this step. Accounting call-site scope.
E-17 Parent gate: parent tombstoned between the gate and the child INSERT Not this step. Tombstone-visibility scope.
E-18 A tombstoned coordinate is reoccupied before the reap Not this step. Reap scope. The count counts the tombstoned row and not the live one, which TestLifecycleScanStore_CountTombstonesByLevel/a_live_artifact_row_is_not_counted,_beside_a_tombstoned_one_that_is covers at the level of the predicate.

Security considerations

# Concern Tests
S-1 Authorization is unchanged and is not weakened Not this step. The gauge makes no authorization decision and exposes no per-tenant value; S-6 carries the disclosure half.
S-2 destructive is a confirmation, not an authorization control Not this step. Repository entry point.
S-3 Deletion is irreversible and the API must say so Not this step. Repository entry point and API contract.
S-4 Input validation Not this step for destructive. The count takes no user-controlled input at all: its one parameter is the context, asserted by TestLifecycleScanStore_CountTombstonesByLevel_Guards.
S-5 No injection surface is added The count binds no value and interpolates none, and the fixtures do the same: internal/accounting/metrics_integration_test.go seeds a tombstoned npm package with two statements rather than concatenating a clock expression into an INSERT. No dedicated test; this is a negative over generated SQL.
S-6 Error messages must not become an existence oracle TestLifecycleScanStore_CountTombstonesByLevel/query_failure_wraps_the_cause_and_names_the_operation, in internal/datastore/lifecycle_scan_integration_test.go. Commit ed3a67cf1 adds a table-name check to that subtest, so the wrap's attribution is pinned: the table name can no longer be deleted from the wrap with the suite still green. The sibling suite's identifier-free assertion has no analogue here. CountTombstonesByLevel(ctx) takes no namespace, so it holds no tenant identifier to keep out of the error text. On the exposition side, TestTombstoneCollector_ExposesOneGaugeLabeledByPurgeScope pins the label set to purge_scope alone, and TestTombstoneCollector_PassesTheCardinalityAudit runs the disallowed-name check that rejects namespace_id, namespace_slug and repository_name.
S-7 Purge failures are operationally visible rather than silent TestTombstoneCollector_APermanentlyStalledPurgeStaysCounted, TestTombstoneCollector_LeaseHolderReportsBothLevels, TestTombstoneCollector_AFailedCountReEmitsTheCachedNumber, TestTombstoneCollector_TwoInstancesExactlyOneEmits, TestTombstoneCollector_ReportsZeroForALevelHoldingNoTombstone

Two rows amended after the first test commit

This branch moves two plan-acceptance rows from the package suite to the composition root. The commit that did so was folded into b17a14c75 by a later rebase, so no commit on the branch carries that change alone. Read these two in place of the rows above that carry the same criterion.

Plan acceptance Tests
The tombstone-count gauge is labeled by level TestIntegration_TombstoneCollectorReachesTheAssembledRegistry adds the composition-root half: the purge_scope label on the assembled registry, under the prefixed family name no package test can assert.
A pod not holding the lease emits no sample, asserted directly Unchanged. The wiring test waits for a held lease rather than asserting the non-holder case, which the package suite owns.

e2e scenario catalogs

No scenario under docs/testing/ is added or affected. That directory holds six files, and none of them mentions a metric, a gauge, or Prometheus.

Diff size

git diff --numstat origin/main...HEAD between the merge base 09b359c74 and this branch's head cc4bcf77, run 2026-08-25T17:05Z: 16 files, 3088 insertions, 63 deletions. Both endpoints are on origin, so a reader can re-derive the figures.

Group Insertions Deletions Files
Tests 2250 0 internal/accounting/metrics_integration_test.go 1334, internal/datastore/lifecycle_scan_integration_test.go 280, cmd/artifact-registry/wire_accounting_metrics_integration_test.go 219, internal/accounting/metrics_test.go 166, internal/datastore/lifecycle_explain_integration_test.go 126, internal/datastore/lifecycle_scan_test.go 125
Production 820 49 internal/datastore/lifecycle_scan.go 351/3, internal/accounting/metrics.go 348/10, cmd/artifact-registry/wire_accounting_metrics.go 56/12, internal/accounting/backlog_collector.go 35/20, internal/metrics/cardinality.go 22/4, internal/datastore/query_names.go 8/0
Docs and recipe 18 14 docs/plans/2026-08-11-s20a-lifecycle-closed-beta.md 7/7, .claude/skills/run-artifact-registry/SKILL.md 6/4, docs/specs/S20-a-lifecycle-closed-beta.md 4/3, docs/dev/observability.md 1/0

Tests are 73 percent of the insertions. The production surface behind them is 820 insertions across six files. The interval-floored cache that ## The count is floored at the sweep interval describes is 123 of those insertions, in internal/accounting/metrics.go, and it is review feedback answered in place rather than a unit of its own. A split does not help here. The change is one collector, one store method, one registration, one cardinality entry, and the test floor those four need. Two of those six production files are consequences of the four units rather than units of their own. internal/datastore/query_names.go carries the query-name constant the store method needs. internal/accounting/backlog_collector.go is comment-only: its 35/20 diff sits inside the countTimeout doc comment, which now describes three reads across two collectors.

Merge order

The dependency on !1867 (merged) is satisfied

This merge request carries a GitLab merge-request dependency on !1867 (merged), S20-A step 17a. !1867 (merged) merged on 2026-08-23T23:30:07Z. The dependency is still recorded on this merge request, and it no longer blocks the merge.

What the dependency was for. The file is internal/metrics/cardinality.go. The duplicated key is purge_scope, which both merge requests wrote. Both write into the same two maps, expectedDistinctValues and closedSetValues. The two insertion points sat far enough apart that git auto-merges both additions with no conflict marker. The result is a duplicate map key, which is a Go compile error rather than a vet finding. The build is what reports it, and nothing reports it earlier. only_allow_merge_if_pipeline_succeeds is on for this project, and !1867 (merged) read not_approved when the dependency was added, so the gate was real rather than a formality.

The clash was not symmetric, and the rebase had one correct outcome. !1867 (merged)'s three-value block is the one that had to survive, because a superset admits every value this gauge emits. Keeping a two-value block instead compiles and passes every test on this branch, because no test here emits unknown. It then rejects the purger's own label the first time a series that carries unknown is gathered, which breaks the purger's audit. The closed-set check runs over gathered series, so a value that nothing emits at boot is not caught at boot either.

What the rebase left. origin/main carries "purge_scope": 3 and the three-value closed set, both from !1867 (merged). This branch adds no map entry of its own. Its whole change to internal/metrics/cardinality.go is three comment hunks, 22 insertions and 4 deletions, recording that the tombstone gauge is purge_scope's second producer and reads the superset !1867 (merged) pinned. The second conflict, in docs/dev/observability.md, resolved the same way: this branch's one catalog row appends after !1867 (merged)'s three.

A correction, carried rather than dropped. The decision to add this dependency rested in part on a failed head pipeline for !1867 (merged): 2782905845, failed at 08:44:27Z. A later measurement overtook that fact. The same pipeline, re-measured at 09:39Z, reads success and finished at 09:26:43Z, and the test:integration: [POSTGRES, 18] job that had failed is green. The decision itself is unchanged.

The plan Status row and !1840 (merged)

The Status table is untouched here, and AGENTS.md:275 is why. That guardrail says a step merge request does not edit the plan file, the Status table included. Parallel step merge requests that edit adjacent rows of one file collide, so the table gets a single writer instead. The three plan edits this branch does carry are named in ## The plan amendments this merge request carries, and none of them is a Status row.

Step 17b's row is recorded in !1840 (merged), the standing plan merge request for this file. !1840 (merged) merged on 2026-08-24T07:11:21Z, and origin/main now carries | 17b | Leased tombstone-count gauge | !1870 |.

The title marker is what finds a step merge request while its Status cell is still empty: (S20-A plan: 17b/21) ends this merge request's title, and one title search over the project returns it.

Both this merge request and !1840 (merged) edit docs/plans/2026-08-11-s20a-lifecycle-closed-beta.md. Their hunks do not overlap: this branch edits line 223, lines 890, 897 and 903, and lines 995 to 997. This branch is rebased onto !1840 (merged)'s merge, so no order between the two is left to force.

Step 16's sweep, and the window before it merges

The sweep that drains the counted tables is !1871 (merged), S20-A step 16, which is open. This merge request asserts no order against it and carries no dependency on it. The condition and its event are written here rather than left to the stack, because a stack can merge out of order.

While !1871 (merged) is open, no component drains the eight tables this gauge counts. Tombstones are written into them already, by the npm and Maven soft deletes and by the three remote evictors. The count is therefore monotonic in that window, and the partial-index plan ## A deviation from the database query guide measures degrades as a table's tombstoned fraction grows. Once !1871 (merged) merges, the sweep drains those tables, and that fraction is bounded by what purge throughput sustains rather than by nothing.

The registration this merge request adds is unconditional whenever both the database client and the Redis client are present. The count therefore runs from the moment this merges, rather than from the moment a purge worker is wired. docs/dev/observability.md carries the reading half of that window, and this section carries the cost half.

A deviation from the database query guide

LifecycleScanStore.CountTombstonesByLevel runs eight COUNT(*) statements and carries no namespace_id predicate. Each parent table is PARTITION BY HASH (namespace_id) with 64 partitions, so one call plans and runs 512 partition scans.

docs/dev/database-query-patterns.md states a partition-key rule and a 100 ms budget. This count meets neither.

Those 512 scans are partial-index scans in the healthy regime alone. The plan is not fixed. The doc comment on countTimeout already carries this hedge for the backlog count. The planner reads the partial index only while tombstones stay a small part of a table. Past a crossover it scans the partition and filters instead, and that costs what the table holds rather than what its tombstones hold. Measured on PostgreSQL 17, against a table that carries this index shape, analyzed, at 200,000 rows:

  • A bitmap scan of the partial index at 0.1%, 5% and 10% tombstoned.
  • A Seq Scan at 25% tombstoned.

The crossover therefore sits between 10% and 25% tombstoned. A stalled purge is what drives a table into that regime, and a stalled purge is the fault this gauge is the control for. Every cost in this section is a healthy-regime cost, so the disclosure understates the exact condition the gauge exists to surface.

The cost is measured. The /db-review-prep query-mode evidence that this merge request owed is note 3721436370. That run seeded 512 namespaces and 64,000 rows per table, with 1.6% of the rows tombstoned, and two passes agreed.

Measurement, one call Value
Planning, all eight statements 678-709 ms
Execution, all eight statements 24-27 ms
Planning, one parent 57.6-137.3 ms
Planning, repositories alone 124-137 ms

repositories alone is past the 100 ms budget of docs/dev/database-query-patterns.md, before a row is read. Against the 5 s that countTimeout allows the caller, the planning total leaves roughly sevenfold headroom rather than an order of magnitude. One parent here plans at 12 to 36 times ADR-007's steady-state figure of 3.85-4.70 ms per parent. Both ends of that multiple divide the measured range by the ADR range, 57.6/4.70 at the low end and 137.3/3.85 at the high one. The earlier "12 to 29" divided both ends by the ADR range's upper bound alone; internal/datastore/lifecycle_scan.go was corrected in 1f6ccab82 and this line now matches it.

Read those numbers as a floor. They come off one seeded database rather than a deployment, and 1.6% tombstoned puts the run inside the healthy regime.

The cadence was the narrow ground here, and ## The count is floored at the sweep interval is what closed it. origin/main already carries deliberate unpruned reads of 64-partition tables, and those run on a background schedule rather than on a scrape. The two schedule defaults are 5 minutes: internal/config/virtualrepositories.go:39 for the health sweep, and internal/config/lifecycle.go:16 for the lifecycle sweep. This count now runs at the second of those two, so it sits with them rather than apart from them.

internal/datastore/lifecycle_explain_integration_test.go pins the fan-out. A namespace_id predicate added later therefore fails loudly. Without that pin, such a predicate turns a fleet-wide total into the total for one namespace, behind a gauge that still reads plausible.

Work item #815 stays the owner of the production measurement. It names the four inputs it needs from outside this repository, and it asks for the crossover point and the cost past it as well as the healthy-regime number.

The count is floored at the sweep interval

Review asked what the count costs per unit time, and the answer needed one input this repository does not hold: the Prometheus scrape interval. Runway sets it centrally, in the OpenTelemetry collector's ArgoCD values, at collectors.daemon.prometheusCR.scrapeInterval: "15s". The collector runs as a DaemonSet, so each pod is scraped by the collector on its own node, once.

At 15 s, the measured 678-709 ms of planning plus 24-27 ms of execution is about 4.8% of one PostgreSQL backend, held continuously. None of it amortizes. docs/dev/database.md requires simple query protocol of all query code for PgBouncer compatibility, so there is no prepared statement and PostgreSQL re-plans all eight Appends on every call. That closes the one configuration change that would have absorbed the planning, and leaves the call rate as the only lever on the cost.

TombstoneCollector therefore caches the count and re-issues it only once the cached value has aged past lifecycle.sweep_interval. At the 5 m default that is about 0.24% of a backend, a 20-fold reduction, and the gauge becomes up to one sweep interval old. That costs its reading nothing: CountTombstonesByLevel already documents that a rule "has to read the trend rather than the value", because a non-zero retention window settles the count at the arrival rate times the window rather than at zero. The interval is the purge sweep's own, so the count refreshes as often as the backlog it counts is drained.

It is BacklogCollector.missingStatistics' shape, ported whole: the cache, a floor on the retry after a failure, and a mutex held across the count.

Two behaviours change, and both are the sibling's.

A failed count now re-emits the cached number instead of dropping the family. The number is one the database returned, only earlier, and dropping it would turn one bad scrape into a blind gauge. What it costs is that a re-emitted sample looks live to Prometheus, so a database outage draws the same flat line a stalled purge draws. The counting tombstones by purge level failed Warn line is the only thing that separates them, and docs/dev/observability.md now tells an alert not to read a plateau alone as a stall. A failure before the first count succeeds has nothing to re-emit and leaves the family absent, which is already what a pod holding no lease exposes.

A failed count is also not retried inside the interval. Without that floor a failing count is re-issued on every scrape, so a database that cannot answer is asked for 512 partition scans at scrape rate rather than once an interval.

Three paragraphs in internal/accounting/metrics.go argued against the cache, the floor and the mutex. Each rested on there being no cache, so each is rewritten rather than edited. The mutex is the clearest case: without a cache it serializes two concurrent scrapes, and with one it coalesces them, which TestTombstoneCollector_ConcurrentScrapesIssueOneCount pins by holding the first count open until the second scrape is provably blocked on the lock.

The cache bounds cost per unit time rather than cost per run. In the degraded regime past the 10-25% crossover a single count can still approach the 5 s countTimeout, on the gather goroutine, holding the whole /-/metrics response. That now happens once per sweep interval instead of once per scrape. It is not removed, and work item 815 keeps the measurement that would size it.

The spec amendment this merge request carries

This merge request edits docs/specs/S20-a-lifecycle-closed-beta.md. It removes the stale ## Open Questions bullet about the tombstone-count metric's registration point, and it adds a ## Resolutions entry in its place.

!1470 (merged) answered that question on 2026-08-12. This diff implements the answer.

The lifecycle-package RegisterMetrics seam was not rejected. The same merged plan puts the purger's three unleased signals on it. That seam carries no mutual-exclusion lease, and a fleet-wide instantaneous count needs one.

The two sentences in docs/plans/2026-08-11-s20a-lifecycle-closed-beta.md that this bullet removal once left stale are already correct on main. !1840 (merged) rewrote both when it merged, and each now states the condition and its event rather than a present-tense fact that this merge request would falsify. One points at the spec's #open-questions anchor and names what happens once !1870 (merged) merges. The other says the same about how many of the spec's open questions Approach answers. Neither needs an edit here.

The plan amendments this merge request carries

This merge request carries three narrow edits to docs/plans/2026-08-11-s20a-lifecycle-closed-beta.md. Each one is an explicit operator override of the rule that a step merge request does not edit the plan.

  1. Step 17b's integration test moves out of internal/lifecycle/metrics_integration_test.go and into internal/accounting.
  2. The plan said that S22's leased-collector step had not merged. That step merged as !1645 (merged) on 2026-08-19, and the plan now says so.
  3. Step 17b's Type: field changes from chore to feat.

A plan sentence this branch falsifies

The plan's run-recipe paragraph ends with "No other step changes either surface." That paragraph names steps 16 and 18 as the two steps that change how the service boots or dispatches. Both also update the run recipe. This branch falsifies the sentence. Step 17b registers a second leased collector at the composition root in cmd/artifact-registry/wire_accounting_metrics.go. It also edits .claude/skills/run-artifact-registry/SKILL.md. The branch re-ran driver.sh smoke at 48 passed and 0 failed.

The correction is routed to !1914 (merged), which is the plan file's single writer today. !1840 (merged) held that role and merged on 2026-08-24T07:11:21Z without making the edit. Its diff carries step 17b's Status row and the two open-question rewrites, and this paragraph appears there only as a hunk context line. a15655a77, the next docs(plans) commit on the same file, touched the Status table alone. The two requests to !1840 (merged) are note 3721145287 and note 3722173745.

AGENTS.md guardrail 4 names a role rather than a merge request, so the correction moves to the role's current holder. !1914 (merged) already rewrites this paragraph to name steps 16, 18 and 19b, so the correction is one merged enumeration of steps 16, 17b, 18 and 19b rather than the sentence agreed on !1840 (merged), which predates step 19b. The request to !1914 (merged) is note 3731890997. This branch does not make the edit, and the sentence stands on this branch and on main.

Other disclosures

The plan's 17a-to-17b dependency edge does not describe this change set. The plan justifies that edge as resting on the registration step 17a establishes. Nothing on this branch touches internal/lifecycle, lifecycle.RegisterMetrics, or anything step 17a creates. The registration lands in internal/accounting and cmd/artifact-registry/wire_accounting_metrics.go instead. The merge-request dependency on !1867 (merged) above is a separate matter, and it comes from the shared purge_scope map key rather than from that edge. The hand-back records this as HB-5.

The spec labels the gauge by level, and the implementation labels it purge_scope. The values match the spec: repository and artifact. The reconciliation drift histograms already own level service-wide with the set {repository, namespace}, and internal/metrics keys closed sets by label name globally. The hand-back records this as HB-7.

A narrowed acceptance criterion. The plan's criterion pairs "a permanently failing purge keeps its repository counted in the gauge" with "re-queued at the bounded once-per-interval cadence". Only the gauge half is claimed here. The re-queue half belongs to step 16's sweep, and the failing-purge framing needs step 15's worker.

Twelve files this change set touches that the plan's step 17b Files: block does not name. The count is against the block as this merge request leaves it, which names four files and reaches all four. This merge request discloses them here and does not add them to the block.

  • .claude/skills/run-artifact-registry/SKILL.md
  • cmd/artifact-registry/wire_accounting_metrics.go
  • cmd/artifact-registry/wire_accounting_metrics_integration_test.go
  • docs/plans/2026-08-11-s20a-lifecycle-closed-beta.md
  • docs/specs/S20-a-lifecycle-closed-beta.md
  • internal/accounting/backlog_collector.go
  • internal/accounting/metrics_test.go
  • internal/datastore/lifecycle_explain_integration_test.go
  • internal/datastore/lifecycle_scan_integration_test.go
  • internal/datastore/lifecycle_scan_test.go
  • internal/datastore/query_names.go
  • internal/metrics/cardinality.go

Related to #611

This is a bot message 🤖 — /smurfit

Edited by Pawel Rozlach

Merge request reports

Loading
Loading