Add wal_receiver recipe for a WAL receiver node outside the Patroni cluster

Summary

Adds a new recipe, gitlab-patroni::wal_receiver, for a WAL receiver: a small machine outside the Patroni cluster that keeps a live copy of the cluster's WAL files with pg_receivewal. If a primary's upload to GCS stops, as on Sep 24 (INC-14518), the receiver still has every change.

• Main issue: #23059: WAL archiving resilience (option C)
• Task: #23272: chef role for the WAL receiver
• Test results the design is based on: follow-up tests, summary and to-do list

Nothing changes for the database nodes. They keep running gitlab-patroni::default. The new recipe only runs on a node whose role lists it.

Why

On Sep 24 (INC-14518), a primary stopped uploading its WAL files to GCS. Those WAL changes existed only on that one machine. If it had been lost, we could not have restored the database to that point.

Today WAL leaves the primary in only two ways:

• the upload to GCS (wal-g), which is what stopped on Sep 24;
• the Patroni replicas. They can be rebuilt or rewound, and they're inside the same cluster.

We need one more copy that doesn't depend on the GCS upload and isn't part of Patroni.

What

A WAL receiver: one small machine per Patroni cluster, outside the cluster.

• It runs pg_receivewal, which copies every WAL change from the primary as it's written, and saves it to the receiver's own disk.
• It isn't a Patroni member, so it can never become a primary or be counted as a replica.
• It follows the primary after a failover, using the rules in "How the receiver decides" below.
• Files are deleted only once GCS has the identical file (see "How the cleanup works").

The recipe goes in gitlab-patroni because it reuses what the database nodes already have: the same secret for the replication password, the same PostgreSQL apt source, the same gitlab-psql user and the same Consul name.

What each file does

File	What it does
recipes/wal_receiver.rb	The new recipe. On the receiver node it: stops chef if Patroni is also in the run list; reads the replication password from the same secret the database nodes use; adds the PostgreSQL apt source and installs only postgresql-client-<version> (no server); creates the gitlab-psql user; writes the password file; creates the data folders; installs the scripts below and the two Ubuntu packages the checksum helper needs (python3-google-auth, python3-crcmod); and sets up walrecv.service and walrecv-cleanup.timer (every 8 hours).
templates/default/walrecv-sticky.sh.erb	The receiver script that walrecv.service runs. It decides which machine to copy WAL from (see "How the receiver decides" below) and runs pg_receivewal.
templates/default/walrecv-cleanup.sh.erb	The cleanup script that the timer runs every 8 hours. It deletes a WAL file only when GCS already has the same file (see "How the cleanup works").
files/default/walrecv-gcs-crc.py	Read-only GCS checksum helper. With wal-g's own key, it reads the size and fingerprint (CRC32C) that GCS stores for each WAL file, and compares them with the receiver's copy. It only reads. Nothing is downloaded, and nothing is written to GCS.
files/default/walrecv-gce-status.py	Read-only GCE check. Given an IP address, it asks the GCE API (with the VM's own token) whether that machine is running or stopped. The receiver uses it to see that an old primary has been fenced.
libraries/wal_receiver_helper.rb	Two small helpers: the Patroni guard (is a Patroni recipe in the run list?) and the Consul name of the primary (master.<service>.service.consul).
attributes/default.rb	The default settings for the receiver (table below). A role can override any of them.
spec/unit/recipes/wal_receiver_spec.rb	Unit tests: only the client package is installed; the password file has the right content, owner and mode and is hidden from logs; the script gets the right Consul name, slot and GCE filter; the service and timer are enabled; the cleanup is in dry-run mode by default; nothing from Patroni is created; the guard detects Patroni; the checksum helper and its packages are installed; and the cleanup looks at files older than 1 day, every 8 hours, without wal-g's extra downloads.
metadata.rb	Version 1.13.0 → 1.14.0, so chef-repo can pin the version with the new recipe.
README.md	A short "WAL receiver" section.

Settings (['gitlab-patroni']['wal_receiver'])

Every setting below can be changed in the receiver's role (per environment and per cluster). If a role doesn't set it, the cookbook default in the table is used.

Setting	Default (cookbook)	Meaning	When to change it in the role
base_dir	/var/opt/gitlab/walrecv	Where the WAL files are stored	Normally never
slot	walrecv	The replication slot on the primary	Always: one per cluster, for example walrecv_main
pgpass_file	/var/opt/gitlab/walrecv.pgpass	The replication login file (mode 600, owner gitlab-psql)	Normally never
gce_name_filter	empty	Which VM names the GCE check looks at	Always: for example patroni-main-v17-.*
keep_days	1	The cleanup only looks at files older than this many days	Only if a cluster needs to keep copies longer
cleanup_dry_run	true	true = the cleanup only reports what it would delete	Set to false once the cleanup output has been checked in gstg
cleanup_parallel	2	How many files the cleanup checks at the same time	For a busy cluster: gprd main should set 4 (and its VM needs at least 4 CPUs)
cleanup_schedule	*-*-* 01,09,17:00:00	When the cleanup runs: every 8 hours, plus up to 15 minutes of random delay	Normally never
connect_options	timeouts	connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=5 keepalives_count=3 tcp_user_timeout=20000: notice a dead primary in about 30 s instead of about 16 min	Normally never

The recipe also uses the existing user, group, secrets, postgresql.version, postgresql.apt, patroni.consul.service_name and patroni.users.replication.username attributes, the same way the database nodes do.

What the receiver script does, step by step

walrecv.service runs /usr/local/sbin/walrecv-sticky.sh. The script runs in a loop:

1. Pick "my node". It reads the IP saved in <base_dir>/node. If the file is empty (first start), it looks up the primary's Consul name (master.<service>.service.consul) and saves that IP.
2. Ask my node what it is. It connects to that IP (waiting up to 10 s for an answer) and asks the database for its identity.
3. If it answers "I'm a primary":
   • it picks the folder for that timeline (<base_dir>/tl<timeline>/), creating it if needed;
   • if the folder has no WAL files yet, it runs the gap check;
   • it starts pg_receivewal with the slot, --synchronous and -n. --synchronous makes it save each change to disk at once and report back to the primary how far it has saved. -n means pg_receivewal never reconnects by itself: the script always decides where to connect;
   • every 10 s, while copying, it checks whether my node has been fenced. If it is fenced and no longer answers as primary, it stops pg_receivewal (force-stop after 5 s). If it still answers as primary, it keeps copying and logs a warning;
   • when pg_receivewal stops, it waits 5 s and goes back to step 1.
4. If it answers "replica / starting up / shutting down" or refuses the connection: my node can't be writing any more. The receiver looks up the Consul name:
   • if Consul has no other primary yet, it waits 5 s and goes back to step 1;
   • if the new machine answers "I'm a primary", it saves the new IP to <base_dir>/node and goes back to step 1;
   • otherwise it waits 5 s and goes back to step 1.
5. If it doesn't answer at all: it checks for a fence (the file <base_dir>/fenced/<ip>, or GCE shows the VM stopped). If fenced, it moves as in step 4. If not, it stays, waits 5 s and goes back to step 1, with no time limit.
6. If the password is refused: it stays on my node, waits 5 s and goes back to step 1.

If the script itself dies, systemd restarts it after 5 s. It picks up the same node from <base_dir>/node.

How the receiver decides which machine to copy from

The receiver remembers one machine ("my node") in <base_dir>/node, and always connects to it by IP address. In a loop, it asks that machine what it is:

The machine answers	What the receiver does
"I'm a primary"	Copies WAL from it into a folder per timeline (<base_dir>/tl<timeline>/)
"I'm a replica", "starting up" or "shutting down"	It can't be writing, so the receiver moves to the new primary
"Connection refused"	Its database is stopped, so the receiver moves to the new primary
No answer	Waits and retries, with no time limit. It moves only once the machine is fenced: a file <base_dir>/fenced/<ip>, or GCE shows the VM stopped

More rules:

• It moves only to a machine that answers "I'm a primary". It finds it through the Consul name.
• It checks for a fence while connected. A frozen primary keeps the connection open, so without this check the receiver would never notice. If pg_receivewal doesn't stop within 5 s, it's force-stopped.
• If a machine is marked fenced but still answers as a primary, the receiver keeps reading and logs a warning, because that machine may still be saving writes.
• Gap alarm: before it starts a new timeline folder, it checks that its slot doesn't start ahead of the files it already has. If it does, it logs GAP ALARM and creates <base_dir>/GAP-<time>, because those files must be filled from GCS.
• Logs go to the journal: journalctl -u walrecv.

Why these rules: a receiver that always follows "whoever is primary now" missed 4,500–5,800 confirmed rows in every split-brain test. With these rules, it missed none (results).

How the cleanup works

When it runs: every 8 hours (01:00, 09:00, 17:00 UTC, plus up to 15 minutes of random delay). That leaves plenty of room: on a gprd main weekday, one run has about 160,000 files, roughly 1–2 hours of work. If a run is still busy, the next one waits for it, so two runs never happen at the same time.

Which files it looks at: complete WAL files in <base_dir>/tl*/ that are older than keep_days (1 day). It never touches .history files, the node file, fenced/ or GAP-*.

What it does with each file:

Group	Which files	What happens
Unfinished	.partial files: the receiver is still writing them	Always kept
Newest file	The newest complete file in the folder the receiver writes into now	Always kept. After a restart, the receiver uses it to know where to continue
Risky	Files where the receiver's copy and GCS's copy can differ: the file written at the moment a new primary took over (read from the .history files), and any file name found in more than one folder (a split brain)	Full compare: download the GCS copy and compare every byte. Identical: deleted. Different: kept
Normal	Every other complete file, which is almost all of them	Fingerprint check: wal-g squeezes the receiver's copy exactly as the primary did before uploading, and the size and fingerprint (CRC32C) are compared with what GCS stores. Nothing is downloaded. Both the same: deleted. Otherwise: kept

Scenarios:

Scenario	Result
Normal day: GCS has the same file	Deleted
GCS doesn't have the file yet (the upload is late, or stopped as on Sep 24)	Kept; checked again at the next run
GCS has a different file with that name	Kept
The file written when a new primary took over	Deleted if identical; otherwise kept
Split brain: the same file name in two folders	Each copy compared on its own: identical deleted, different kept
No writes for a day	The newest file is kept, so the receiver restarts correctly
Any error (GCS unreachable, wrong key, wal-g failed)	Kept; checked again at the next run
The receiver's wal-g doesn't match the primaries'	Every file kept ("check failed"), and the alert fires (#23274)
Dry run (cleanup_dry_run: true, the default)	Only logs "would delete"; nothing is deleted

Why every 8 hours: a file is always kept for at least 1 day; running more often just deletes it sooner after that day. Example, for a file that arrives on Monday at 10:00:

• once a day at 01:00: it's still too young on Tuesday at 01:00, so it's deleted on Wednesday at 01:00, after about 39 hours on disk;
• every 8 hours: it's deleted on Tuesday at 17:00, after about 31 hours on disk.

So the receiver needs a smaller disk. For gprd main on a weekday, that's about 12 TB instead of about 26 TB.

Files that are kept are checked again at every run, and deleted as soon as GCS has the same file. A file that really differs from GCS after a split brain is never deleted automatically, because it may be the only copy of writes the new primary never got. A person removes it after the incident (runbook step in #23275).

Testing

• Real chef run on the db-benchmarking test receiver node, in local mode from this branch, with only gitlab-patroni::wal_receiver in the run list:
  • the first run applied 21 of 31 resources; a second run changed nothing;
  • walrecv.service started, found the primary through Consul, and streamed WAL into its timeline folder, with complete files arriving;
  • its slot was active on the primary;
  • the password file had the right line, owner gitlab-psql, mode 600;
  • the cleanup ran in dry-run mode and exited cleanly;
  • the new cleanup (fingerprint check, every 8 hours) passed every test: decisions, real deletion, duplicate names, failures, a restore proof, speed and a receiver restart (test results);
  • with gitlab-patroni::default added to the run list (why-run mode), chef stopped with gitlab-patroni::wal_receiver must not run on a node that also runs Patroni, and nothing changed.
• Unit tests (pipeline): all 10 wal_receiver tests pass; the whole suite passes (102 examples, 0 failures).
• Test Kitchen (same pipeline): all existing suites pass on Ubuntu 20.04 and 22.04, so nothing changes for the database nodes. There's no Kitchen suite for wal_receiver yet: the real run on db-benchmarking above covers it.

Not in this MR

These come as separate MRs:

• chef-repo: the receiver roles, and the pin to 1.14.0;
• config-mgmt: the receiver VMs;
• chef-repo: the receiver's slot in the Patroni roles. It's merged last: a slot with no receiver makes the primary keep WAL;
• runbooks: alerts, and runbooks and the hand-over script.

Rollback

Remove gitlab-patroni::wal_receiver from the receiver node's role, or pin the previous version (1.13.0). The database nodes don't use this recipe, so they're not affected either way.