## Macro item 2: WAL archiving resilience (P0) - https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23026#note_3915201882 ## Problem On 2026-09-24 the boot disk of the gprd main primary, `patroni-main-v17-01`, stopped completing I/O from about 22:43 to 23:23 UTC. The data disk was healthy, so Postgres kept committing, but WAL archiving to GCS stopped for the whole 40 minutes. This mattered because archiving was the only other copy of 01's WAL: - the replicas stopped streaming from 01 at 22:49:56 - 03 was promoted at 23:20:29 from the last WAL it had (22:49:55) - 01 kept committing until 23:23:14 After streaming stopped at 22:49:56, 03's only other source of WAL was GCS, which had nothing newer than 22:43. So 03 could not get past 22:49:55. Everything 01 committed in the last \~33 minutes stayed on 01 only and did not make it into production. If archiving had kept running, those writes would have been in GCS within about a minute of being committed (`archive_timeout = 1min`). ## Root cause **1. Everything the archive command needs to start is on the boot disk.** All gprd Patroni clusters (main, ci, sec, registry) archive with the same command, wal-g 3.0.5: ``` archive_command = /opt/wal-g/bin/archive-walg.sh %p ``` which runs: ``` /usr/bin/envdir /etc/wal-g.d/env /opt/wal-g/bin/wal-g wal-push $1 &>>/var/log/wal-g/wal-g.log ``` | Needed by the archive command | Path | Disk on 01 | |-------------------------------|------|------------| | Script | `/opt/wal-g/bin/archive-walg.sh` | boot (`/dev/nvme0n1p1`) | | `envdir` | `/usr/bin/envdir` | boot | | wal-g binary | `/opt/wal-g/bin/wal-g` | boot | | Config + GCS key | `/etc/wal-g.d/` | boot | | wal-g log | `/var/log/wal-g/wal-g.log` | log disk (`/dev/nvme2n1`) | | WAL files | `/var/opt/gitlab/postgresql/data17/pg_wal/` | data disk (`/dev/nvme1n1`) | Sources: [archive-walg.sh](https://gitlab.com/gitlab-cookbooks/gitlab-walg/-/blob/master/files/default/archive-walg.sh), [gitlab-walg attributes](https://gitlab.com/gitlab-cookbooks/gitlab-walg/-/blob/master/attributes/default.rb), [gprd-base-db-patroni-main-v17.json](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/master/roles/gprd-base-db-patroni-main-v17.json), `df` on 01. **2. One archive call hung and blocked all archiving.** WAL is written in 16 MB files, and each file is archived by one call of the command above. Postgres runs one archive call at a time and only starts the next one when the previous call returns. - 22:43:07: last line in the wal-g log before the gap. The file it archived was `00000002001B755500000055`, and it reached GCS. - The next file, `00000002001B755500000056`, was not archived until after the disk recovered. The wal-g log has no lines at all between 22:43:07 and 23:23:19, although the log disk was healthy. wal-g logs `Files will be uploaded to storage` early in every run, right after loading its config ([wal_uploader.go#L87](https://github.com/wal-g/wal-g/blob/v3.0.5/internal/databases/postgres/wal_uploader.go#L87)), so no archive call got that far during the stall. - 22:47:10: kernel hung-task report for the boot disk journal: `task jbd2/nvme0n1p1-:1566 blocked for more than 122 seconds`. - 23:23:20: Postgres log, the only archiver line that day: `FATAL archive command was terminated by signal 3: Quit` `The failed archive command was: /opt/wal-g/bin/archive-walg.sh pg_wal/00000002001B755500000056` - 23:23:31: first successful upload of `…56` in the wal-g log. Pending WAL on 01 (`pg_archiver_pending_wal_count`) went from 5 at 22:43 to 5,931 at 23:24, the last sample before Postgres on 01 was stopped. **3. The alert for this fired but didn't page.** `walgBackupDelayed` (no WAL archived for 15 minutes) fired at 23:04 and cleared at 23:22. It is `severity: s3` and only posts to Slack ([rule](https://gitlab.com/gitlab-com/runbooks/-/blob/master/mimir-rules/gitlab-gprd/autogenerated-gitlab-gprd-walg-backups-alerts.yml)). `pg_archiver_pending_wal_count` kept reporting from 01 throughout the stall, so an alert on pending WAL growth would also have worked. ## Still to confirm **Which exact step the archive call hung in.** Candidates, all needing the boot disk: - starting the script or `envdir` - loading the wal-g binary - reading `/etc/wal-g.d` - the first GCS request (DNS and TLS files under `/etc`) The logs can't narrow this down. No process stack was captured on the day, and the kernel stopped logging hung tasks after reaching its limit of 10. This decides what an option like A (below) would need to move, so it will be confirmed with a reproduction. ## Plan ### 1. Confirm the root cause - [x] Map the archive path and the disks it depends on - [x] Collect evidence from 01 (`wal-g.log`, `postgresql.csv`, `kern.log`) - [ ] Save a copy of 01's Sep 24 `wal-g.log.1` before logrotate removes it - [ ] Reproduce on `patroni-amtest-v17` (db-benchmarking) and capture the stack of the stuck archive process - [ ] Post the confirmed root cause here ### 2. Evaluate options "Boot disk" below means the primary's OS disk, mounted at `/` (on 01: `/dev/nvme0n1p1`). It is the disk that stalled on Sep 24. The data disk and log disk are separate and were fine. Each option is tested with the same freeze test: freeze the primary's boot disk under write load and check whether the WAL written during the freeze still ends up somewhere outside the primary. **Pass** = every WAL file up to the primary's last commit is copied off the primary while the boot disk is frozen. **Option A: make the archive command not need the boot disk** - What: move everything `archive-walg.sh` needs (the script, the wal-g binary, `/etc/wal-g.d` config and GCS key) to the data disk, so archiving keeps working when the boot disk stalls. - Why it might help: on Sep 24 the data disk and the network were fine; only the boot disk stalled. - To find out: - After moving those files, does archiving keep running with the boot disk frozen? - Does wal-g still touch the boot disk anyway, for example system files it needs to reach GCS (DNS config `/etc/resolv.conf`, TLS certificates under `/etc/ssl`) or `/usr/bin/envdir`? **Option B: `archive_mode = always` on replicas** WAL leaves the primary in two independent ways: streamed to the replicas over the network (how they stay in sync), and archived to GCS by wal-g (the backup copy). On Sep 24 archiving stopped at 22:43, but streaming kept working until 22:49:56, when Patroni on the replicas disconnected them because there was no leader. After that, WAL went nowhere. - What: replicas also upload the WAL they receive to GCS, not only the primary. - Why it might help: if the primary's archiving breaks while streaming still works, GCS still gets the WAL through the replicas. - Limit: a replica can only upload WAL it has received. On Sep 24 the replicas had WAL up to 22:49:55, and the lost writes are everything after 22:49:56. So on its own this option would not have saved any of the lost data. - To find out: - With the primary's archiving stuck but streaming working, does WAL keep reaching GCS through the replica? - With the replica also cut off from the primary (as on Sep 24), confirm it stops too. - Primary and replicas would upload the same files to the same GCS path. Does wal-g handle that safely? **Option C: `pg_receivewal --synchronous` on a separate node** - What: `pg_receivewal` is a Postgres tool that connects to the primary the same way a replica does, receives the WAL stream, and saves the WAL files to its own disk. It runs on a separate VM, does not run a database, and is not managed by Patroni. - Why it might help: on Sep 24 the replicas stopped receiving WAL because Patroni on the replicas disconnected them (no leader), not because 01 could not send. 01's log shows the replicas closing their connections at 22:49:57, and its data disk and network were fine. A receiver that Patroni does not control would not have been disconnected, so it would likely have kept receiving 01's WAL after 22:49:56. That is the WAL that was lost. - `--synchronous`: the receiver writes each piece of WAL to its disk as soon as it arrives and tells the primary straight away, instead of waiting for a full 16 MB file. On its own this does not make commits wait for the receiver; that only happens if it is also listed in `synchronous_standby_names` (synchronous replication, macro item 5). - Limits: it is a new component to run and monitor on every cluster. Its WAL files are on the receiver's disk, not in GCS, until something uploads them. - To find out: - Does it keep receiving WAL while the primary's boot disk is stuck? - After a failover, how does it find and connect to the new primary? - It should use a replication slot, so the primary keeps WAL for it if it is down. How much WAL can pile up on the primary, and what limit (`max_slot_wal_keep_size`) keeps the primary's disk safe? - How do the received files get to GCS, and how would we use them in a restore? * [ ] Run the freeze test for each option on `patroni-amtest-v17` and record the results here * [ ] Review the results with DBRE and agree on the option (or combination) ### 3. Implement the chosen option - [ ] Open follow-up MRs / issues for the agreed option - [ ] Validate with the freeze test in gstg - [ ] Roll out to gprd: main, ci, sec, registry ## Acceptance criteria - The exact step where archiving blocked on Sep 24 is confirmed and posted - Options A, B and C tested with the freeze test, results posted - An option agreed with DBRE and documented here - The agreed option rolled out to all gprd Patroni clusters, and passing the freeze test ## Related - Incident: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23018 - Incident review: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23026 - Macro item 1, watchdog: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23046 - Macro item 4, alerts: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23045. `walgBackupDelayed` fired at 23:04 but is s3; `pg_archiver_pending_wal_count` kept reporting from 01 during the stall. - Half-dead primary runbook: https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29936