docs(lifecycle): required alerts for the purger and where they land

Summary

This merge request writes down the alerts the lifecycle purger needs. It wires none of them.

docs/dev/alerting.md is a new page, and it carries a ### Required alerts section for the purger under ## Lifecycle purger. The section carries four alerts. Each one names its series, its condition, and the readings that are not a fault. The section also states four rules that hold across all four alerts:

1. The test the severity split follows.
2. How to size for: against the shape of the expression, rather than against the number in its window.
3. The selector rule, which is that no rule adds a target or topology label to its selector beyond env.
4. The file in gitlab-com/runbooks the rules are created in.

docs/specs/S20-a-lifecycle-closed-beta.md gains a named pointer to that section, which makes the alerting catalog the single home for the list. The same page also carries storage accounting's three reconciliation paging alerts, moved out of docs/dev/storage-accounting.md byte for byte, so one page now holds every required alert this service has. docs/dev/observability.md and docs/dev/storage-accounting.md each keep their ### Required alerts heading and carry a pointer to the new page, and docs/dev/README.md gains an index row for it. internal/datastore/lifecycle_scan.go records in the doc comment of CountTombstonesByLevel that the counts run in sequence and that the first failure aborts the whole map. That is the mechanism the narrower on the first alert depends on.

Acceptance criterion 4 is not met, and this is deliberate. The criterion asks that the rules exist in gitlab-com/runbooks and that each rule links a runbook. No merge request exists in that repository, so no rule runs. Until one lands, the purger has no alerting, and the new section is a specification rather than a description of anything that runs. The merge request in that repository waits on one measurement, so this is a sequencing decision rather than an open question. ## What remains names that measurement and says why this unit cannot take it.

Six files, 274 insertions(+), 22 deletions(-), measured with git diff --stat origin/main...HEAD at fef44b233. Of those, 259 insertions are the new docs/dev/alerting.md, and 216 of the 259 are the purger section moved out of docs/dev/observability.md unchanged apart from four repointed sentences. This is under the 500-line threshold, so the description carries no size justification.

Acceptance criteria

#	Criterion	Verdict	Evidence
1	A required-alert list for the purger, in the shape S22 uses	Met, in a third home	### Required alerts under ## Lifecycle purger in docs/dev/alerting.md. The criterion names docs/dev/observability.md or the S20-A spec and no third home, so this is a deliberate deviation. See ## Where the list lives, and why it is not where criterion 1 says
2	Each alert names series, condition, and non-fault readings	Met	All four alerts carry the three parts. Both arguable severities are marked as arguable
3	The discovery-scan failure has an alert, or the list records why it does not	Met, through both arms	Alert 2 sits on kind="lifecycle:tombstone-sweep". The list records the one arm that alert misses, and why no rule selects it
4	The rules exist in gitlab-com/runbooks and each links a runbook	Not met, deliberate	No merge request exists in that repository. See ## What remains
5	A reader auditing GA alerting against the specs finds the list	Met	docs/specs/S20-a-lifecycle-closed-beta.md points at the list from the end of its ### Observability section, and docs/dev/README.md, docs/dev/observability.md and docs/dev/storage-accounting.md each carry a path to docs/dev/alerting.md

On criterion 3, the arm the list records rather than alerts on: SweepWorker.probeScope stops at the first purge level that answers a row. In a namespace that holds an earlier level's tombstone, a broken later level is never probed. The tick records status="ok", so alert 2 stays silent, and the failure surfaces on kind="lifecycle:purge" instead. No alert selects that kind, because it folds a failed scan in with four other readings that need opposite responses.

Where the list lives, and why it is not where criterion 1 says

Acceptance criterion 1 of issue #947 (closed) names two homes, verbatim: "The S20-A spec, or docs/dev/observability.md, carries a required-alert list for the purger, in the shape ### Required alerts uses for S22." docs/dev/alerting.md is neither of those two, so this merge request deviates from that criterion as written. The deviation is deliberate, and it is the answer to @suleimiahmed's review on docs/dev/observability.md.

He asked for an alerting document "where all required alerts can be catalogged (not just for lifecycle)". Two required-alert lists existed before this change, and neither one linked the other: the purger's, added by this branch, and storage accounting's three reconciliation paging alerts in docs/dev/storage-accounting.md. An engineer auditing alerting coverage who opened either file alone saw a complete-looking list. That is the failure issue #947 (closed) was filed against, one level up, so honoring criterion 1 literally would have reproduced the problem the criterion exists to prevent.

Criterion 5 is the one that carries the purpose, and it is met more fully than before: "A reader auditing GA alerting against the specs finds this list." Four paths now carry that reader to it. The docs/dev/README.md Project Guides table has a row for alerting.md. docs/dev/observability.md keeps its ### Required alerts heading and points at the new page. docs/dev/storage-accounting.md does the same, and its Quick reference table gains a Required alerts row. docs/specs/S20-a-lifecycle-closed-beta.md points at ../dev/alerting.md#lifecycle-purger.

Issue #1003 goes stale on this merge, and that is recorded rather than left to be found. Its second site table names five docs/dev/storage-accounting.md lines, and four of them, 1323 to 1326 at origin/main, are inside the block this change moves. The fifth, line 1401, sits in ## Known gaps and does not move. Note 3769868328 on that issue gives the new file and line for each of the four, and states that the quoted text is unchanged.

What remains

Acceptance criterion 4 remains, and this merge request uses Related to rather than Closes for that reason.

The four rules must be created in mimir-rules/runway/artifact-registry-gke.yml in gitlab-com/runbooks, with one runbook annotation on each rule. That file does not exist today, so the operation is a create and not an edit. The path without the .yml is a directory of autogenerated files that carry a DO NOT EDIT header, and the generator rewrites them. The sibling to copy is mimir-rules/runway/pmdb-dist-svc.yml, a hand-written file that lives beside its own generated directory of the same name.

That merge request opens only after one measurement, the count by (env) (runway_lb_request_count{type="artifact-registry-gke"}) query against the mimir-runway datasource that ## Not measured, and silent when wrong also names. That one query settles the env value every rule must carry. A rule file transcribed before it is answered can carry the wrong value, select nothing, and report nothing about selecting nothing. The query needs Grafana access this unit does not have, so it is not this unit's to run. Its absence is not a blocker on this merge request. docs/dev/alerting.md now records the same decision, in the paragraph that opens The rules are not wired, and the order is deliberate rather than an omission.

Nothing schedules the wiring

Work item 354, "Grafana dashboards and alerting for the .com closed beta", is open, and a reader looks to it for this wiring. This unit read that item through the API on 2026-09-01. While its only alerting phase reads as it does, that phase covers none of the four rules. The phase is ### Phase 4: AR-specific alerts, a single unchecked bullet, whose subjects are authz denial spikes, buffered-counter loss rates, usage-data drops and npm deadline kills. The words lifecycle, sweep, purge, purger and tombstone occur zero times in that item's description, read case-insensitively across the whole of it. It carries no assignee and no milestone while it stands as written. Once that phase is widened to name the purger, work item 354 becomes the item these four rules are transcribed under. This subsection is then the part to re-read. While the phase is not widened, the four rules belong to no scheduled item.

One open merge request in this project reads the other way, and the disagreement is recorded here rather than settled. !2034 states in its own description that "alert wiring is out of scope and belongs to #354 (closed)". The reading above comes from work item 354's own description, which is the text this unit measured. Which of the two is right is not decided here, and this merge request proposes no action on !2034.

Follow-up work items

Both are open, and the new section links each one from the place that raises it.

• Work item 1010 carries a fifth candidate alert, which is deferred rather than dropped. A purge_outcome="parent_pinned" that recurs across re-queues needs a rule that reads persistence instead of a rate.
• Work item 1011 carries the on-call rotation question, which is what the "severity with a rotation" column of the list asks. With a rotation, the sweep-liveness rule becomes s2 with a pager label, and the other three stand as proposed.

Not measured, and silent when wrong

Two inputs to these rules are guesses today, and each one fails without a signal.

• No one has confirmed that any gitlab_artifact_registry_* series reaches the Mimir tenant. A rule that selects nothing is silent, not broken. It never fires, and nothing reports that it never fires.
• The env value is not established, gprd against production. This query settles it: count by (env) (runway_lb_request_count{type="artifact-registry-gke"}) against the mimir-runway datasource. A rule that carries the wrong value selects nothing, in the same silent way.

Every threshold and every window in the list is a proposal with no production baseline behind it. The first pass against real series will move them.

One correction this unit made to its own brief

The brief asked for RunwayJobFailure and its s4 beside the sweep-liveness alert, as the nearest precedent that disagrees with the proposed severity. RunwayJobFailure is kept, and so is its s4. It is not the nearest precedent. GitalyShardWeightsAssignerStale is a time-since-last-success deadman, which is the shape of the sweep-liveness alert, and it carries s2 with a pager. That precedent argues the severity up rather than down. The list now names both, and says which way each one argues.

Merge order and conflicts

Merge request	Shared files	Effect
!2040 (merged)	internal/datastore/lifecycle_scan.go and docs/dev/observability.md	Conflicts on the Go file only. Both branches rewrite the same three-line doc comment. docs/dev/observability.md does not conflict
!1973 (merged)	docs/specs/S20-a-lifecycle-closed-beta.md	Merged on 2026-08-31. This branch is rebased onto it and the spec pointer applies cleanly in its new context
!2078 (merged)	docs/dev/observability.md	Merged on 2026-09-01 at 08:56Z. Its ### Reading the container-remote cache events is in this branch's base, and the rebase kept both sections, that one first

Resolution is this branch's three lines verbatim, because the rewrite here drops the table count that !2040 (merged) renumbers, so nothing is left in that sentence for !2040 (merged) to change. The two merges still need a deliberate order rather than a race.

The !2078 conflict resolved the way that row predicted, and it is settled rather than pending. The two sections are independent, so the rebase kept both and ordered them with the container-remote section first. That resolution is in a0650501c.

The !2040 overlap in docs/dev/observability.md is not a conflict. git merge-tree --write-tree --name-only --no-messages between !2040 (merged) at 24ee1fd19 and this branch at fef44b233 names internal/datastore/lifecycle_scan.go and nothing else. The same command was run against every other open merge request that touches one of this branch's six files, and the move added no conflict. !2123 (merged), !2162 (merged), !2163 (merged), !2166 (merged), !2169 (merged), !2187 (merged), !2193 (merged) and !2103 (merged) all merge cleanly with this branch, and the six of those that touch docs/dev/storage-accounting.md include every merge request that could have collided with the block moved out of it. !1598, !1069 and !2020 (merged) do conflict, and each of them conflicts with origin/main on the same file already, so that is their own rebase debt rather than this branch's doing.

This branch writes no count that !2040 (merged) renumbers, so the new section does not go stale when that merge request lands. The regex this list prescribes, ".+_select_tombstone_count", also matches the queryMavenPackagesSelectTombstoneCount that !2040 (merged) adds, so the two agree rather than compete. It also depends on nothing !1973 (merged) introduced. That merge request has since merged, and it added the ### Required alerts heading to docs/dev/storage-accounting.md that this branch now moves into docs/dev/alerting.md. Both headings sit in that one file, under different ## parents, and no-duplicate-heading is configured siblings_only, so the linter permits them. What the linter does not police is the rendered anchor: two identical headings render as #required-alerts and #required-alerts-1, so every link this branch writes targets an ## anchor, #lifecycle-purger or #storage-accounting, instead.

Governing ADRs

Nine cited ADRs were read in full. Three are genuinely engaged, and each one agrees with the change: 010_data_retention, 022_namespace_decoupling, and 025_garbage_collection. No ADR governs where alert rules live, the s1 to s4 severity vocabulary, the alert_type label, or paging.

The local docs/adr/ mirror reports STALE, with five unsynced commits on 007 and 021. The three drifted hunks of 007 were read at the handbook source. They touch no tombstone, purge, partition, or observability text, so the conclusion holds. One internal ADR never mirrors to docs/adr/, and this review did not read it. A conflict with that one ADR is therefore not excluded.

Testing

The change adds no Go code beyond a doc comment on an existing exported method, so it adds no Go test file.

• markdownlint-cli2: 0 errors.
• vale --minAlertLevel=error: silent.
• lychee --offline --include-fragments: Passed on all six changed files. The two new anchors, alerting.md#lifecycle-purger and alerting.md#storage-accounting, resolve, and no observability.md#required-alerts reference is left anywhere in the tree. .gitlab/ci/docs.gitlab-ci.yml runs this check in the blocking docs:lint job.
• golangci-lint: the golangci-lint pre-commit hook passed on the committed tree. A direct golangci-lint run --new-from-merge-base=origin/main ./... could not run in this checkout: the binary on PATH is 2.12.2 and .golangci.yaml on origin/main enables exhaustruct_v5, which that version rejects with unknown linters. The config is unchanged by this branch, so the mismatch predates it.
• Comment caps: 3 lines against the 3-line cap for an exported method, and 159, 160, and 152 columns against the 160-column lll limit, the second line sitting at the limit exactly rather than under it. Measured at fef44b233 with awk 'NR>=381 && NR<=383 {print length($0)}' internal/datastore/lifecycle_scan.go. The third line lost 5 columns, because docs/dev/alerting.md is shorter than the path it replaced.
• Every metric name, label key, label value, Go symbol, and log string in the new section was read back against the tree.
• Both linked work items were fetched through the API, because nothing in CI checks them.
• The branch review resolved all 89 open merge-request head refs and left none unchecked. Nine overlap these three files, six merge cleanly, and three conflict.
• Nothing here has been executed. No pass has run these expressions against a live datasource, and none has put a draft rule file through validate-alerts or mimirtool rules check. Five passes have read this section; zero have run it.
• The two gitlab-com/runbooks claims were read from that project's API at master, not from a local checkout. RunwayJobFailure carries s4, alert_type: symptom, no for: key, and no environment selector. GitalyShardWeightsAssignerStale carries s2, pager: pagerduty, alert_type: cause, and for: 1h. Of the three hand-written rule files under mimir-rules/runway/, only pmdb-dist-svc.yml filters on env, and all 28 of its selectors read env="gprd".

What moved after the review request

The review request went to @suleimiahmed at 2026-09-01T02:52Z, when the head was the commit now replayed as 0141484a5. @suleimiahmed reviewed at 2026-09-01T09:31Z and requested changes. The head is now fef44b233. The branch has since been rebased onto origin/main at 9431b68ae, which is why no sha this section named before is still on the branch. The rebase replayed all ten earlier commits byte-identically, checked with git range-diff --creation-factor=100, which returned = on every row.

Five commits landed across that span. The first four touch docs/dev/observability.md and no other file. f924175ed adds two paragraphs: the first says the rules are not wired and that the order is a sequencing decision rather than an omission, and the second says the section has been read and not run and names the two checks that close that gap. f0f8bf3ae qualifies alert 1's bullet marker by lifecycle.retention_window, so the marker states a condition that holds at both settings rather than at the default alone. d0e60dcc3 gives alert 1's window paragraph the crossing points alert 3's paragraph already carried, and the retuning instruction that keeps this rule's for: longer than its window at every admitted interval. 2a534cb73 scopes the sweep-liveness absent-arm sentence to the life of the exporting process, and writes the second reading of a firing arm.

The fifth, fef44b233, is the move that answers @suleimiahmed's review, and it is the one commit here that touches more than one file. It creates docs/dev/alerting.md, moves both required-alert lists into it, and repoints the five sites that named the old locations.

No alert changed in any of the five: no severity, no for: value, no range window, no alert_type, no label and no expression. The storage-accounting list moved byte for byte, checked with diff, and the purger list moved with four sentences repointed, each of which had named a metric-catalog row as "above" while that row stays in docs/dev/observability.md.

The AppSec review cleared a commit that four rebases have now replayed as cc96f492b. The sha the clearance names, c45e56cd5b, no longer resolves in this repository, so cc96f492b is the honest way to name it. A clearance covers the sha it was given, so the honest claim is about what the later commits touch. Eight of the nine commits after cc96f492b touch docs/dev/observability.md and nothing else. The ninth, fef44b233, touches four more Markdown files and one Go file. Its whole Go change is one line of comment text inside the existing doc comment of CountTombstonesByLevel, replacing one document path with another, measured with git show fef44b233 -- internal/datastore/lifecycle_scan.go as 1 insertion(+), 1 deletion(-). No executable Go statement, no configuration file, no dependency file and no CI file has moved since cc96f492b.

Review guidance

The expensive failures here are the factual claims about mechanism. The section asserts a great deal about code it does not change. A wrong claim of that kind is invisible at review time, and it becomes expensive later. A reviewer has the option to read those claims first, against internal/lifecycle/sweep.go, internal/lifecycle/purge.go, and internal/datastore/lifecycle_scan.go.

Five passes ran over this branch, and what each one found is the useful part.

The first validation pass found four blockers, all of them mechanism claims that did not hold against the code. A second pass re-checked each correction and returned advisories only. Those two read the prose against the Go code, and they were right about it.

Nobody read the rules against PromQL semantics, the multi-pod series shape, or the alertmanager routing until the third pass. Every pass that has looked at those has found defects: a branch review found four, a merge-readiness pass found three more, and that pass's own triage then found a defect in one of its fixes. That is a coverage gap rather than proof the rules cannot be written down, and the last two rounds are the ones that closed it.

The four the branch review found were rules that, written exactly as the section stated them, either never fire or fire permanently.

• The tombstone-gauge rule matched no series at all. Its two operands carry disjoint labels, so an unqualified and returns empty at every evaluation and reports nothing about the fact.
• The sweep-liveness rule fired forever on any fleet holding more pods than its window holds ticks, because the counter resolves to one series per pod and a process exports a counter it once incremented for the rest of its life.
• The narrower's rate window sat at one sweep interval, where a single zero evaluation restarts the two hours the for: needs.
• The tombstone-gauge rule named no expression form at all, and "failing to trend toward zero" is three rules with three different false-positive profiles.

Those were closed, and the three the merge-readiness pass found afterwards were of the same family. Both written-out expressions aggregated the env label away, so the rules would have fired and reached no team channel — the alertmanager route matches env and team together, and an earlier round had added only team. Alerts 2 and 4 had no written expression, and the only shape on offer resolved per pod, so no pod's series held the condition long enough for its for: to complete. Alert 1's for: 2h restarted at every lease handover.

The last commit is the one worth reading if you read only one. The bridge written for that third finding was itself a sweep interval too short, because a successor whose first count fails emits nothing for that interval on top of the lease vacancy. The omitted term correlates with the fault the rule exists for: a stalled purge is often a database that cannot answer, which is exactly what makes that first count fail. The bridge failed in the case the alert is for.

The section also gained the two things a transcribed rule needs and did not have: the title annotation, which validate-alerts treats as fatal where runbook is only a warning, and the team: artifact_registry label, without which all four rules validate, deploy, fire, and reach no team channel.

Labels

Issue 947 carries type::feature. This merge request carries type::maintenance and documentation.

docs/dev/labels.md says that an MR overrides a label from its linked issue when the label does not match the actual work of the MR. This merge request ships no capability. The alerts do not exist after it merges, which is exactly what criterion 4 records.

The override matches how the project labels this work. Of the 100 most recently merged merge requests, 20 carry a docs title, and 19 of those 20 carry type::maintenance with documentation.

Paired-documentation obligations

Three pairings are stated here rather than left silent.

• e2e scenario catalogs. The guardrail asks a feat or fix merge request to update docs/testing/ or to say why not. This is a docs change with no behavior change, so the obligation does not fire.
• Configuration reference. The diff touches no internal/config/**, no proto/artifactregistry/config/**, and no config.example.yaml.
• Bruno collection. The diff touches no api/openapi/**.

Process deviation

A documentation author wrote this unit, rather than the test-first cycle of the repository. The unit introduces no Go code. The only Go change is a doc comment on an existing exported symbol, and the issue card records that no Go test file is needed. This unit skipped no test.

Related to #947 (closed)

This is a bot message 🤖 — /smurfit