## Summary The lifecycle tombstone sweep re-enumerates the namespace keyset from the start on every tick. A tick that runs out of its budget makes no progress at all: the next tick starts again from the same end of a stable ordering, so the namespaces past the cut point are never reached. This is the shipped and intended behaviour. It was decided deliberately for closed beta, and the spec states it. This issue records the point at which that decision stops being the right one, and the arithmetic for deciding what replaces it. ## Why this is filed now rather than fixed now The sweep is introduced by !1871 (S20-A step 16). While that merge request is open no sweep runs anywhere, and the starvation this issue describes is unreachable for that reason alone; once it merges, the behaviour below is live and the arithmetic here is what decides when it stops being acceptable. The starvation is unreachable at closed-beta fleet size, and the alternatives all cost more than the problem is worth at that size: a durable cursor needs a schema migration, and a randomised start needs wrap-around and is a spec change of its own. The decision was to ship restart and close the gap in the spec, the configuration text and the observability instead. ## Which namespaces starve, and why it is not an arbitrary set Namespace ids are UUIDv7, generated at `internal/datastore/ids.go:16-18`. The walk orders by `namespaces.id ASC`, so the ordering is creation order. The starved tail is therefore **the most recently onboarded tenants**. A new customer added past the cut point is never swept until the fleet shrinks or the interval grows. ## The arithmetic One tick costs approximately `8 * N_live * t`, plus a negligible enumeration term. - Eight probes per namespace: one repository-scope level and seven artifact-scope levels, which are the eight levels `DefaultPurgeLevels` names. - A namespace holding no tombstone still pays all eight. - Frozen, deleted and purged namespaces pay zero probes, because `sweepNamespace` skips a namespace that is not write-serviceable before any probe runs. So `N_live` is the write-serviceable count, not the total. At the default five-minute `sweep_interval`, a tick stops completing at: | Probe cost `t` | Crossover `N_live` | Practical ceiling | | --- | --- | --- | | 0.5 ms | 74,800 | about 37,000 | | 1 ms | 37,400 | about 19,000 | | 3 ms | 12,500 | about 6,000 | The practical ceiling is about half the crossover in each row, because a tick that only just fits leaves no headroom for a slow page. `sweep_interval` is validated below one hour, so raising the interval buys a bounded amount of room. **Past roughly 150,000 to 440,000 write-serviceable namespaces, no value of `sweep_interval` inside that cap achieves full coverage**, whichever end of the `t` range holds. Page size does not move any of this. Probes outnumber enumeration queries by about 800 to 1, so the enumeration term is noise. ## The two numbers this rests on are not in the repository Neither the probe cost `t` nor the write-serviceable fleet size is recorded anywhere in the tree, and neither was measured. Both are observable in production with no new code: - `gitlab_artifact_registry_jobs_duration_seconds{kind="lifecycle:tombstone-sweep"}` gives the whole tick's duration, which divided by the walked count gives `8t`. - The sweep's own per-tick log line carries `namespaces_walked`, `namespaces_skipped`, `namespaces_probed` and `purge_jobs_enqueued`. Decide this against a measured `t` rather than against the estimates above. ## Condition and event: when this becomes live While the write-serviceable fleet stays below the crossover for the measured `t`, every tick completes, the cursor resets, and the sweep has no coverage gap — the behaviour and the spec agree, and nothing here applies. Once the fleet crosses that point, three things happen together and all are visible without new instrumentation: 1. The tick begins failing on its budget, which surfaces as a terminal failure on the sweep kind. 2. The per-tick counters show `namespaces_walked` short of the fleet, which is the direct read of how much was missed. 3. The most recently onboarded tenants stop being swept, and their tombstoned rows stay on disk. The interim lever is a larger `sweep_interval`, up to the one-hour cap. That is a mitigation, not a fix, and it runs out. ## Options, if this is picked up These were priced during the step-16 review and are recorded here so the work does not start from scratch. - **Random start with wrap-around.** Cheapest behaviour change. Gives a probability of coverage, not a bound: a namespace is missed for `k` consecutive ticks with probability `(1 - f)^k`. Needs wrap-around to be correct at all — without it, the walk covers only `[p, end]` and the *oldest* namespaces become almost unreachable, which is strictly worse than restart. The sibling `remote.HealthSweepWorker` shuffle at `internal/remote/healthjob.go:163` is **not** a transferable precedent: it enumerates the whole fleet into memory before shuffling, and its shuffle exists for upstream-host jitter rather than coverage. - **Resume in memory with wrap-around.** Adds a cursor field on the worker plus a concurrency guard, since the retry ladder can run attempts of the same worker concurrently. A per-pod cursor is the per-instance execution ADR-011 line 54 names as a thing to avoid, and every deploy resets it. - **Resume durably with wrap-around.** The only option with a deterministic fleet-wide bound, and the only one under which a truncated tick's remainder really is picked up next time. Costs a schema migration and a persisted cursor. - **Reshape the walk to enumerate tombstones rather than namespaces.** The discovery indexes are partial on `soft_deleted_at IS NOT NULL`, so a walk of them costs the tombstone count rather than eight times the namespace count. This contradicts the spec's design section, which states the sweep is a loop over namespaces, so it is a spec rewrite. ## Related - !1871 — S20-A step 16, which ships the sweep and the restart behaviour. - #611 — S20-A: lifecycle closed-beta implementation. - #641 — the same class of question for the remote health sweep enumeration, which is a different worker and not a duplicate of this one.