## The gap `grep -ic alert` over `docs/specs/S20-a-lifecycle-closed-beta.md` returns **0**. S22 states its obligation. `docs/dev/storage-accounting.md` carries a `### Required alerts` section naming three paging alerts, and [#435](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/435) and [#515](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/515) hand those three to [#354](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/354) by name. S20-A names no alert anywhere. So the purger's alerting is not an owed obligation waiting for an owner. **None is specified.** This issue is where that gets specified and shipped. ## Why the absence is the dangerous shape An alerting audit run against the specs before GA finds S22's three, finds nothing for lifecycle, and reads as complete. The purger is the one component that removes customer data. It runs by default: `cmd/artifact-registry/wire_lifecycle.go:91` registers it, `internal/config/lifecycle.go` defaults `sweep_interval` to `5m`, and `config.example.yaml` defaults `retention_window` to `0s`. There is no `enabled` flag. ## Depends on work item 935, which is a hard prerequisite [#935](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/935) is what puts the three purger metric families on `/-/metrics`. `lifecycle.RegisterMetrics` has no production caller today, so no rule written now can select any of them. **Merge order:** work item 935 lands first. Until it does, the only purger series a rule can read are the jobs-foundation ones, `gitlab_artifact_registry_jobs_processed_total` and `gitlab_artifact_registry_jobs_duration_seconds` at kinds `lifecycle:purge` and `lifecycle:tombstone-sweep`, plus the `gitlab_artifact_registry_lifecycle_tombstones` gauge, which reaches the registry by a different path through `accounting.NewTombstoneCollector`. ## What already exists, and what it is not `docs/dev/observability.md` carries extensive rule-authoring caveats inside the purger metric descriptions. They constrain rules nobody has been asked to write, and they are the best starting material for this issue: - On `purge_outcomes_total`: "an alert on `rate(...{purge_outcome="error"})` alone will page on any deploy that overlaps a purge", because `jobsriver.Client.Shutdown` force-cancels River's running jobs and records `error` on the in-flight row. - On `lifecycle_tombstones`: "An alert on this gauge must allow for more than one scrape interval, and must not read a plateau alone as a stalled purge." A re-emitted cached sample looks live, so a database outage draws the same flat line a stalled purge draws. - On `lifecycle_tombstones`: a namespace that is not write-serviceable holds a fixed floor under the fleet-wide value, and no label isolates it. That floor is a freeze rather than a purge that cannot complete. - On `purge_outcomes_total`: `parent_pinned` "names the class of refusal and not any one cause of it ... a refusal that keeps recurring is a defect and a single one is contention." Only persistence across re-queues separates them. These are notes to an author. They are not a required-alert list, and no document turns them into one. ## The one failure no counter sees `purgeLevel` returns as soon as a level's discovery scan fails, so that level records no increment on `purge_outcomes_total` and no sample on either histogram. `docs/dev/observability.md` states the consequence: "a level whose scan fails every tick looks exactly like a namespace holding no tombstones." Two signals carry it and neither is a purger metric family: the `lifecycle purge discovery scan failed` Error line, and `gitlab_artifact_registry_jobs_processed_total{kind="lifecycle:purge",status="error"}`. Whichever alert set this issue settles on has to cover that case explicitly, because the obvious rules do not. ## Candidate alerts Starting material rather than a decision. Each is drawn from the caveats above, and the trade-off named beside it is what the design work has to resolve. 1. **Purge progress stalls.** `gitlab_artifact_registry_lifecycle_tombstones` failing to trend toward zero at `retention_window: 0s`. The catalog gives three readings of a plateau — a failed count re-emitting its cache, a frozen namespace's floor, and a genuine stall — and a rule has to separate them or accept that it cannot. 1. **Discovery scan failing every tick.** The invisible case above. Reachable through the jobs-foundation counter and the Error line, not through the purger families. 1. **Sweep liveness.** The `lifecycle:tombstone-sweep` kind failing to record a successful execution within a bounded multiple of `sweep_interval`, on the shape S22's reconciliation trigger-liveness alert uses. 1. **Purge failures.** `purge_outcomes_total{purge_outcome="error"}`, with the deploy-overlap false positive handled rather than ignored. 1. **Recurring `parent_pinned`.** Persistence across re-queues is the only separator between a defect and contention. Whether all five page, or some only warn, is part of the work. ## Relationship to work item 354 Work item 354 owns the metrics-catalog entry and the rule files in `gitlab-com/runbooks`, and it is where these rules land. Its own scope enumerates per-route and per-format error rates, throughput, and DB and Redis saturation, and it names no background-job subsystem. This issue supplies what work item 354 has no way to derive: which purger failures warrant a page, and which readings of each signal are false. Work item 435 plays the same role for S22's three. ## Acceptance criteria 1. The S20-A spec, or `docs/dev/observability.md`, carries a required-alert list for the purger, in the shape `### Required alerts` uses for S22. 1. Each alert names its series, its condition, and the readings that are **not** a fault. 1. The discovery-scan failure has an alert, or the list records why it does not. 1. The rules exist in `gitlab-com/runbooks` and each links a runbook. 1. A reader auditing GA alerting against the specs finds this list. ## Related - [#935](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/935) — prerequisite. Puts the three families on `/-/metrics`. - [#354](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/354) — owns the metrics-catalog entry and the rule files. - [#435](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/435) — the same role for S22's three required alerts. - [#728](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/728) and [#937](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/937) — four tables whose tombstones no scan reads. `docs/dev/observability.md` records that a counted row from them would never drain, which is a standing floor any progress alert has to tolerate until they are fixed. Related to https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/611