Loading
Zoekt observability: the ORR runbook pages and the alerts index
Adds a runbook page for each of the 8 Zoekt alerts added in merge request 2 of this stack, plus an index at docs/zoekt/README.md. The runbook: annotation on every one of those rules points here, so scripts/validate-alerts resolves each link once this lands.
- Each page states what the alert measures, what it does not, the first queries to run, and where to route when the signal is not actionable from Zoekt itself.
- Three panels referenced by the storage and node pages are empty on gprd and gstg today because both environments pin gitlab-exporter 16.8.0 and the metrics arrived in 16.9.0. The affected page says so and routes the on-caller elsewhere.
- Prose only. No rule, dashboard or test changes.
Reviewer focus: whether the triage steps are the ones an on-caller who has never touched Zoekt would actually follow.
Review order
Merge request 3 of 3 splitting !11489 (closed), which was declined as too large to review.
| Order | Merge request | Target | Contents |
|---|---|---|---|
| 1 | Zoekt observability: dashboard | master |
Dashboard and its two reducer guards |
| 2 | Zoekt observability: alert rules | jmason/zoekt-obs-1-dashboard |
8 alerts, jsonnet + generated YAML + promtool tests |
| 3 | this one | jmason/zoekt-obs-2-alerts |
9 docs/zoekt/ pages, prose only |
@johnmason to leave feedback.
🤖 Verification notes (for agents / deep readers)
markdownlint-cli2 --config .markdownlint.ymlover the 9 changed pages — 0 issues in 0 files.scripts/validate-alertsexits 0, with everyrunbook:annotation from merge request 2 resolving to a page added here.- The split is lossless: every one of the original merge request's 20 file blobs is byte-identical at the top of this stack, and
git diffover those paths between the original branch and this one is empty.
What I did not verify
- The pages have not been walked through against a real incident; the triage steps are derived from the metric definitions and the Rails code, not from a post-incident review.