## What is not measured `LifecycleScanStore.CountTombstonesByLevel` in `internal/datastore/lifecycle_scan.go` runs eight `COUNT(*)` statements with no `namespace_id` predicate. Each of the eight parent tables uses `PARTITION BY HASH (namespace_id)` with 64 partitions. Nothing prunes, so one collection plans and runs 512 partition scans. The eight parents are `repositories`, `container_images`, `container_remote_manifests`, `maven_versions`, `maven_remote_versions`, `npm_packages`, `npm_versions`, and `npm_remote_versions`. `docs/dev/database-query-patterns.md` states a partition-key rule (line 16) and a 100 ms budget (line 11). This count meets neither, and the doc comment on the method records the deviation. The 512 scans now carry one measured figure, and that figure is a floor. A `/db-review-prep` query-mode run seeded 512 namespaces and 64,000 rows per table, with 1.6% of the rows tombstoned. The eight statements planned in 678-709 ms and executed in 24-27 ms, stable across two passes. `repositories` alone planned in 124-137 ms, which is already past the 100 ms budget before a row is read. That run is note 3721436370 on https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1870. ADR-007's partitioning amendment is the only other published figure for a comparable Append, and it was taken on a different query shape. One parent here plans at 12 to 29 times its 3.85-4.70 ms steady-state figure per parent. ## The large-backlog regime is one open half The eight statements are answerable from partial indexes that hold tombstones and nothing else. The planner reads those indexes only while tombstones stay a small part of a table. Past a crossover it scans the partition and filters instead, and the count then costs what the tables hold rather than what they have tombstoned. Measured on PostgreSQL 17, against a table that carries this index shape, analyzed, at 200,000 rows: - A bitmap scan of the partial index at 0.1%, 5% and 10% tombstoned. - A `Seq Scan` at 25% tombstoned. The crossover therefore sits between 10% and 25% tombstoned. That measurement covers one table at one row count, so it does not fix the crossover for the eight parents at production scale. A stalled purge is what drives a table into that regime. A stalled purge is also the fault this gauge is the control for, so the count turns table-sized exactly when its number is worth reading. A count that then exceeds the 5 s `countTimeout` allows drops the whole metric family, and a dropped family reads the same as a pod that holds no lease. ## Whether this count needs a cost control is answered This item no longer owns that question. The deciding input was the Prometheus scrape interval, and it is 15 seconds, which `## Four inputs that this repository does not hold` records with its source. At that interval the seeded figures put the count at roughly 4.8% of one PostgreSQL backend, held continuously, so the cadence problem `## The ground for this item` states is real rather than hypothetical. [!1870](https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1870) takes all three controls together. While that merge request is open, `TombstoneCollector` still runs the count on every scrape and holds no floor, no mutex and no coalescing. Once it merges, the collector caches the count, re-issues it only once the cached value has aged past `lifecycle.sweep_interval`, floors a failed count's retry at the same interval, and holds a mutex across the count so two concurrent scrapes issue one set of statements between them. **The sibling's shape ports once there is a cache.** This item was filed saying it does not, and the reasoning was sound at the time: `BacklogCollector.missingStatistics` floors a failed count by handing back its cache, and a collector that caches nothing has nothing to hand back, so the same floor would hold the family absent for the whole window. The cache is what removes that premise, which is why !1870 takes the cache and the floor together rather than either alone. The same applies to the mutex: without a cache it serializes two concurrent scrapes, and with one it coalesces them. What that costs the gauge is recorded where an operator reads it, in `docs/dev/observability.md`: the value is up to one sweep interval old, and a failed count re-emits the cached number, so a database outage draws the same flat line a stalled purge draws. ## The ground for this item An unpruned read of a 64-partition table is not new in this repository. Seven merged examples exist, in `internal/datastore/namespace_encryption_keys.go`, `internal/datastore/namespace_encryption_keys_rotation.go`, `internal/datastore/container_remote_repositories.go`, `internal/datastore/maven_remote_repositories.go`, and `internal/datastore/npm_remote_repositories.go`. This count is not the first of its kind, and this item makes no such claim. The cadence is the part with no precedent here. The seven merged examples run on a background schedule. Two of those schedules default to 5 minutes: the health sweep at `internal/config/virtualrepositories.go:39`, and the lifecycle sweep at `internal/config/lifecycle.go:16`. This count ran on the lease-holding pod once per Prometheus scrape when this item was filed. The deployment scrapes every 15 seconds, which put it about 20 times more often than the closest merged precedent. [!1870](https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1870) floors it at `lifecycle.sweep_interval`, which is the second of those two schedules, so once that merge request merges this count runs at the same cadence as the closest precedent rather than 20 times it. The cadence gap this paragraph describes is what that change closes; the row counts and the crossover it does not. ## Four inputs that this repository does not hold 1. ~~**The Prometheus scrape interval and the scrape timeout.**~~ The interval is **15 seconds**. Runway sets it centrally rather than per service, at `collectors.daemon.prometheusCR.scrapeInterval` in `services/otel-collector/values.yaml` of `gitlab-com/gl-infra/argocd`. The collector runs as a DaemonSet, so the collector on each node scrapes that node's pods and no replica pair doubles the rate. Nothing in this repository states it: `.runway/values.yaml` opts a ServiceMonitor in and sets no timing, and `.runway/fairway.yaml` declares the metrics port alone. The scrape timeout is still unread, and it bounds how long one collection may hold the `/-/metrics` response. 1. **An `EXPLAIN (ANALYZE, BUFFERS)` on production-scale data.** `docs/dev/database-query-patterns.md:164` asks for a dataset representative of production. No such dataset exists here, and `docs/dev/database-migrations.md:257` records that Database Lab is not yet available for Artifact Registry. The `/db-review-prep` query-mode run on https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1870 seeded 512 namespaces and 64,000 rows per table, at 1.6% tombstoned, so its timings are a floor rather than an answer. 1. **The namespace and tombstone counts that the closed beta expects.** ADR-007's benchmark assumes 500,000 namespaces and 1.6M blobs. Nothing in this repository states what the closed beta expects. 1. **The CPU budget of the pod under a 512-scan scrape.** `.runway/fairway.yaml` sets `requests.cpu: "100m"` and `limits.cpu: "200m"` for the service container. ## When to do this The closed-beta launch on 2026-09-07 creates the data that this measurement needs. Take the measurement after that date. ## What closes this item Two measurements and one decision close this item. 1. **The healthy-regime cost against real data, at the real scrape interval.** The seeded floor is 678-709 ms of planning and 24-27 ms of execution, at 1.6% tombstoned. Production row counts, catalog size, connection churn, and concurrency all sit outside that run. 1. **The crossover, and the cost past it.** Find the tombstoned fraction at which the planner drops the partial index, for each of the eight parents, at production row counts. The seeded run puts that fraction between 10% and 25% on one table at 200,000 rows, and nothing pins it for the eight. Then measure one collection past that point, and compare it with the 5 s `countTimeout` allows. 1. ~~**The decision on a cost control.**~~ Made, in [!1870](https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1870), and `## Whether this count needs a cost control is answered` carries it. The two measurements above stay open, and what they now size is a control that exists rather than one that does not. Then record both numbers in the doc comment on `CountTombstonesByLevel`, record the decision beside them, and remove that comment's pointer to this item. If both measured costs fit the budget, no code change is needed. If either does not, three remedies were worked and declined when the count landed, and each one changes what the gauge delivers: - ~~Cache the count on an interval, the way `BacklogCollector.missingStatistics` caches the orphan count.~~ Taken, in [!1870](https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/1870). It trades the count's own cost for staleness, which one sweep interval bounds while the count succeeds and nothing bounds during an outage. The two below stay available and neither is taken. - Count fewer levels. `docs/specs/S20-a-lifecycle-closed-beta.md` requires both levels, so this breaks an acceptance criterion. - Count per namespace. The spec's own sizing argues against it, and at the namespace counts that ADR-007 assumes it is worse than the current shape. Each of the three needs a decision before it is taken. ## Precedent for the shape of this item Two items were filed for the same reason, one statement each that the project reasoned about and never measured: - https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/564 - https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/668