**Parent:** [#23059](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059) **Background:** we chose option C for WAL archiving resilience. Each Patroni cluster gets a **WAL receiver**, a small machine that keeps a live copy of the cluster's WAL files. Then writes aren't lost when the upload to GCS stops, as on Sep 24. Test results: [summary](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_4000115706). ### What A role per cluster, for example `gprd-base-db-wal-receiver-main`. It is built on the base role, **without** the gitlab-patroni cookbook, which would install and start Patroni. ### The role sets up 1. **Consul names:** the recipe `gitlab-server::systemd-resolved_consul`, so `master.patroni-<cluster>.service.consul` resolves. Check the network rules are loaded; on one amtest node they weren't. 2. **Postgres 17 client tools:** `pg_receivewal` and `pg_waldump`. 3. **wal-g with the same settings as the database nodes** (`/etc/wal-g.d/env`), so the receiver can upload its files to GCS in a hand-over ([4h](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_3999827336)) or to fill a gap ([4j](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_3999298029)). No backups and no automatic uploads. The receiver must use **the same wal-g role as its cluster** (gprd: `role[gprd-walg-3_0_5-2204]`) **and copy the cluster's `walg_attr`** (`walg_compression_method: brotli`, `gcs_normalize_prefix`, `gcs_max_retries`), so that a wal-g upgrade always covers the receiver too, and the cleanup's checksum check matches ([Vamshi's review](https://gitlab.slack.com/archives/C02K0JTKAHJ/p1793086796600929?thread_ts=1791215302.246629&cid=C02K0JTKAHJ)). The amtest and gstg receiver roles already do this. 4. **The replication password file:** - The password is in the encrypted chef secrets file the database nodes already use: `gs://gitlab-gprd-secrets/gitlab-patroni/gprd.enc` (KMS key `gitlab-secrets/gprd`; gstg: `gitlab-gstg-secrets/gitlab-patroni/gstg.enc`), under `gitlab-patroni → patroni → users → replication → password`. - Read it the same way as [gitlab-patroni recipes/default.rb](https://gitlab.com/gitlab-cookbooks/gitlab-patroni/-/blob/master/recipes/default.rb), with `get_secrets(...)` from `gitlab_secrets`. - Today nothing writes this file for a non-Patroni machine. The cookbook only writes the superuser's file ([recipes/patroni.rb](https://gitlab.com/gitlab-cookbooks/gitlab-patroni/-/blob/master/recipes/patroni.rb)), and on database nodes Patroni writes the replication file itself. - No new access is needed: the VM service account can already unlock the key (config-mgmt `environments/gprd/main.tf`, `google_kms_key_ring_iam_member "secrets"`). - Write `/var/opt/gitlab/walrecv.pgpass` with one line, `*:5432:*:gitlab-replicator:<password>`. Owned by `gitlab-psql`, mode 600, marked sensitive. A password change is picked up on the next chef run. 5. **The sticky receiver, as a system service** ([S1–S11](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_4051377725)): - It remembers its machine in a file, and after a drop reconnects to it **by address**. - It moves to the new primary only when its machine answers as a replica or "starting up", its database is stopped, or it is **fenced**: a file `fenced/<ip>` written by the hand-over runbook, or GCE shows the machine stopped. - The Consul name `master.patroni-<cluster>.service.consul` is only used to find the new primary. - It checks for a fence while connected, and force-stops `pg_receivewal` if it doesn't stop within 5 s. - If a machine is marked fenced but still answers as a primary, it keeps reading and logs a warning *(not tested yet)*. - **No time limit** ([S4c](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_4051377725): with one, it lost 5,785 writes). - **Gap alarm:** before streaming into a new folder, it checks that the slot doesn't start ahead of what node 06 already has ([S10](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_4051377725)). - A folder per timeline, so it continues by itself after a failover ([4l](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_3999396189)). - Connection string: `host=<node address> port=5432 user=gitlab-replicator target_session_attrs=read-write connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=5 keepalives_count=3 tcp_user_timeout=20000`. It notices a dead primary in ~30 s instead of 16 min ([4k](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_3998998559)). - systemd, starts at boot, `Restart=always` ([4m](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_3999571165), [S2](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_4051377725)). 6. **A primary that is on but stuck:** detected by the write probe ([#23045](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23045)). The receiver moves on once it is fenced ([S4b](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_4051377725)). 7. **Logs to the system journal,** not to the WAL disk. When that disk was full, the log stopped too ([4m](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23059#note_3999571165)). 8. **Cleanup:** every 8 hours, it deletes WAL older than 1 day (`keep_days`), **only if GCS has the same file**. Normal files are checked by size and CRC32C fingerprint, without downloading; files at a timeline switch and duplicate names are downloaded and compared byte by byte. It always keeps `.partial` files, the newest file of the current folder, and any file that differs. It stops with an error if wal-g's compression isn't brotli. gprd main sets `cleanup_parallel: 4` and needs at least 4 CPUs. Details and tests: [gitlab-patroni!210](https://gitlab.com/gitlab-cookbooks/gitlab-patroni/-/merge_requests/210#note_4091894395). Scripts: `walrecv-sticky.sh`, `walrecv-s.service`, `gce-status.py`. They are available on my laptop. ### Done when In gstg, the receiver runs as a service and receives WAL with no gaps, including after a switchover.