**Parent:** [#23059](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059) **Background:** we chose option C for WAL archiving resilience. Each Patroni cluster gets a **WAL receiver**, a small machine that keeps a live copy of the cluster's WAL files. Then writes aren't lost when the upload to GCS stops, as on Sep 24. Test results: [summary](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_4000115706). ### What The alerts in the runbooks repo (`libsonnet/alerts/`, `mimir-rules-jsonnet/`): [runbooks!11944](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11944). | Alert | Metric | Fires when | Why | |---|---|---|---| | Receiver not connected | `pg_replication_slots_active{slot_name=~"walrecv(_.*)?"}` on the primary | `0` for 5 min | Covers down, wrong password, network or stuck ([4i](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_3996638310), [4m](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_3999571165)) | | Too much WAL kept for the receiver | `pg_replication_slots_restart_lsn_bytes{slot_name=~"walrecv(_.*)?"}` on the primary | above ~25% of **that cluster's own** limit (gprd main 3,640 GB, ci 6,400 GB, sec 1,600 GB, registry 510 GB) | The primary's disk fills; past the limit the slot is lost ([4i](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_3996638310)) | | Receiver disk filling | `node_filesystem_avail_bytes`, `/var/opt/gitlab` on the receiver | under 15% free | A full disk stops the receiver ([4m](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_3999571165)) | | Gap alarm | a `GAP-*` marker file written by the receiver | any | Node 06 is missing WAL; fill it from GCS ([S10](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_4051377725)) | | Fenced but still primary | the receiver's warning in its log | any | The fence didn't work, and the old primary may still be writing | | Cleanup keeps files | `walrecv_cleanup_files{result=~"check_failed\|different"}` on the receiver | more than 10 | The receiver's wal-g no longer matches the primaries', so nothing is deleted and the disk fills ([comment](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23274#note_4089616306)) | | Cleanup not succeeding | `walrecv_cleanup_last_success_timestamp_seconds` on the receiver | no successful run for 9 hours (it runs every 8); a `0` (never succeeded) is ignored | Nothing is deleted, so the disk fills ([comment](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23274#note_4089616306)) | | Cleanup never succeeded | `walrecv_heartbeat_timestamp_seconds` without a successful cleanup | the receiver has run 9 hours with no successful cleanup | Same as above, for a new receiver | | Receiver metrics missing | `walrecv_heartbeat_timestamp_seconds` on the receiver, and `node_uname_info{fqdn=~"walrecv-.*"}` | the heartbeat is more than 10 minutes old, or missing on a `walrecv-*` node | Without the receiver's metrics, the gap, fence and cleanup alerts can't fire ([Vamshi's review](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11944#note_4123175604)) Each alert links to its runbook (5/6). All thresholds (25%, 15%, 5 min) are still to decide; they weren't tested. ### Follow-ups for "Receiver metrics missing" From [Vamshi's review](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11944#note_4123894640). Both are non-blocking. 1. **node_exporter down:** when node_exporter stops, both the heartbeat and `node_uname_info` go stale, so the alert can't fire, although its page lists this as a cause. Use `up` in the second part instead. It reports `0` for a failed scrape, and it carries `fqdn` (checked on gprd: job `scrapeConfig/monitoring/prometheus-agent-node-exporter-gce`): ```promql max by (env, environment, fqdn) (up{%(selector)s, job=~".*node-exporter.*", fqdn=~"walrecv-.*"}) unless on (fqdn) walrecv_heartbeat_timestamp_seconds{%(selector)s} ``` 2. ~~**Timing on the page:** for a stale heartbeat, the query already waits 600 seconds, and `for: 10m` adds 10 minutes more. So it fires about 20 minutes after the last write, and 10 minutes for a missing heartbeat. The page's "Fires after 10 minutes" should say that.~~ Done in [runbooks!11944](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11944) (1c5581b6e): the page now says about 20 minutes for a stale heartbeat, 10 for a missing one. ### Done when The alerts are live in gstg and fire in a test (receiver stopped, disk filled).