Publish WAL receiver and cleanup metrics through node_exporter

Summary

The WAL receiver (gitlab-patroni::wal_receiver, version 1.14.0, gitlab-patroni!210) now publishes a few numbers about itself, so the receiver alerts can use them. Without these numbers, 4 of the 7 planned alerts can't fire, because today this information is only in the receiver's log.

• Main issue: #23059
• Task: #23274, alerts for the WAL receiver (alert plan)

Nothing changes for the database nodes. Only receiver nodes run this recipe.

Words used below

Word	What it means	More detail
WAL	The files where Postgres records every change. The primary uploads them to GCS, and the receiver keeps its own copy	—
Receiver	A small separate machine that copies WAL from the primary all the time	gitlab-patroni!210, "What the receiver script does, step by step"
Timeline	A number that goes up each time a new primary takes over. The receiver keeps each timeline's WAL in its own folder	gitlab-patroni!210, "What the receiver script does, step by step"
Slot	A bookmark on the primary that tells it "the receiver has copied up to here, keep everything after this"	#23273
Fenced	Someone has marked the old primary as finished: its VM is stopped, or a file fenced/<its IP> exists on the receiver	gitlab-patroni!210, "How the receiver decides which machine to copy from"
Cleanup	A job on the receiver that, every 8 hours, deletes WAL files older than 1 day that GCS already has, and keeps everything else	gitlab-patroni!210, "How the cleanup works", and why it works this way

How the numbers reach the monitoring

1. Every VM already runs node_exporter, which gives Prometheus the machine's numbers.
2. node_exporter also reads any .prom text file in its text-file folder (/opt/prometheus/node_exporter/metrics/). Chef already uses this for chef-client.prom.
3. node_exporter runs as prometheus, and only prometheus or root can write in that folder. So the receiver and the cleanup, which run as gitlab-psql, write their files into their own folder, /var/opt/gitlab/walrecv-metrics, and chef links them into node_exporter's folder.
4. The scripts write a temporary file and rename it, so node_exporter never reads a half-written file.

The numbers

File	Number	What it is, in simple words	Normal value	What a bad value means	Alert
walrecv.prom	walrecv_streaming	Whether the receiver is copying WAL right now	1	0 for more than a moment: it's not copying, for example while it waits for a stuck old primary or after an error	— (alert 1 covers this from the primary's side)
	walrecv_gap_markers	How many gaps the receiver has found. When it starts copying a new timeline, it checks that the primary will send WAL from exactly where its own files end. If the primary would start later (for example, the slot was lost because the receiver was off too long), the WAL in between is missing from the receiver. It then writes a marker file GAP-<time> and logs GAP ALARM	0	1 or more: the receiver's copy has a hole. The missing files are normally in GCS; the runbook says how to copy them over and then delete the marker	4. Gap alarm
	walrecv_fenced_but_primary_last_timestamp_seconds	The last time the receiver found a machine marked as fenced that still answered as a primary. The receiver then keeps reading from it, because it may still be writing, and logs a warning	0 (never happened)	A recent time: the fence wasn't done properly, and the old primary may still be accepting writes	5. Fenced but still primary
	walrecv_heartbeat_timestamp_seconds	The time the receiver script last checked in. It does this about every 10 seconds	A time within the last ~20 seconds	Not moving: the receiver script is dead or hung	— (alert 1 also catches a dead receiver)
walrecv_cleanup.prom	walrecv_cleanup_files{result="deleted"}	How many files the last cleanup run deleted, because GCS has the same file. In dry run, how many it would delete	Any number	—	—
	walrecv_cleanup_files{result="unfinished"}	Files kept because the receiver is still writing them (.partial)	A few	—	—
	walrecv_cleanup_files{result="newest"}	The newest finished file in the folder the receiver writes into. Always kept, because after a restart the receiver uses it to know where to continue	1	—	—
	walrecv_cleanup_files{result="timeline_switch_different"}	Files from the moment a new primary took over, or with the same name in two folders, where the full byte compare found they differ from GCS. Kept, because after a split brain they may be the only copy of writes the new primary never got	0	Above 0 after a failover: check the incident; the runbook says when they can be removed	—
	walrecv_cleanup_files{result="not_in_gcs"}	Files GCS doesn't have yet, for example because the upload is late or stopped. Kept, and checked again at the next run	0 or a few	Growing run after run: the primary's upload to GCS may have stopped	—
	walrecv_cleanup_files{result="different"}	Files whose fingerprint (size and checksum) doesn't match the copy in GCS. Kept	0	More than a handful: something is wrong; see alert 6	6. Cleanup keeps files that are already in GCS
	walrecv_cleanup_files{result="check_failed"}	Files the cleanup couldn't check, for example because GCS was unreachable, or because wal-g on the receiver no longer matches the primaries'. Kept, and checked again next time	0	More than a handful: usually wal-g on the receiver and on the primaries differ; nothing gets deleted, and the disk fills	6
	walrecv_cleanup_last_success_timestamp_seconds	The time the cleanup last finished successfully. A run that stops with an error doesn't update it	Within the last 8 hours	Older than 9 hours: the cleanup is failing or not running, so nothing is deleted and the disk fills	7. Cleanup failed or didn't run
	walrecv_cleanup_last_run_timestamp_seconds	The time the cleanup last ran, successful or not	Within the last 8 hours	Recent, while "last success" is old: the cleanup runs but keeps failing	7
	walrecv_cleanup_dry_run	Whether the cleanup is only reporting (1) or really deleting (0)	1 until checked in gstg; 0 on amtest	—	—

What each file does

File	Change
recipes/wal_receiver.rb	Creates metrics_dir; creates both .prom files empty if they don't exist yet (so node_exporter never sees a broken link before the first cleanup run); links them into node_exporter's text-file folder when that folder is configured
templates/default/walrecv-sticky.sh.erb	Writes walrecv.prom. The receiver's decisions are unchanged; any error while writing the file is ignored
templates/default/walrecv-cleanup.sh.erb	Writes walrecv_cleanup.prom at the end of a run, and when it stops because wal-g's compression isn't brotli (then "last success" keeps its old value)
attributes/default.rb	New setting metrics_dir (default /var/opt/gitlab/walrecv-metrics). Like every receiver setting, a role can change it
spec/unit/recipes/wal_receiver_spec.rb	Tests for the folder, the files, the links and the scripts' metrics folder
metadata.rb, README.md	Version 1.14.0 → 1.15.0; one README line

Testing (db-benchmarking amtest, test node 06, chef local mode)

Test	Result
node_exporter shows the receiver's numbers through the links	✅ walrecv_streaming 1, heartbeat moving
A new receiver before its first cleanup run	✅ no read error (node_textfile_scrape_error 0)
A gap marker added, then removed	✅ walrecv_gap_markers 1, then 0
A fence file on the current primary	✅ the receiver warned and kept reading; the timestamp was set; still streaming
A normal cleanup run	✅ last run and last success set; counts correct
A cleanup with the wrong compression	✅ stopped with exit 1; last run updated; last success kept
The receiver script paused for 25 s	✅ heartbeat aged to 31 s, then fresh again after resuming
Chef doesn't overwrite the live metric files	✅ "up to date"
Unit tests and Test Kitchen (pipeline)	✅ 103 tests, 0 failures; all Kitchen suites pass on Ubuntu 20.04 and 22.04

Not in this MR

• The alert rules themselves: a runbooks MR (#23274).
• The runbook pages and the hand-over script: a runbooks MR (#23275).

Rollback

Pin the previous version (1.14.0). The receiver keeps working without the metric files; only the alerts that use them stop firing.