# Production Change ## Change Summary The gprd Patroni Sec DR **archive** replica (`postgres-sec-dr-archive-v18-01-db-gprd`, cluster `gprd-patroni-sec-archive-v18`) has been down since **2026-11-04 04:25 UTC**. Patroni reports `Standby Leader | start failed`, and PostgreSQL refuses connections on port 5432. **Root cause (from `postgresql.csv`):** the data volume filled up during WAL replay. The startup process was restoring WAL from the archive (timeline 2, segments `0000000200172B...`, restored normally) and then hit: ``` 2026-11-04 04:25:06.856 GMT ... FATAL,53100,"could not extend file ""base/16406/912949364.69"": No space left on device",,"Check free disk space.",,,"WAL redo at 172B00/2006410 for Heap/INSERT+INIT ..." ... "startup" 2026-11-04 04:25:06.883 GMT ... LOG,00000,"shutting down due to startup process failure" ... "postmaster" ``` Disk state at the time of diagnosis: - `/var/opt/gitlab` (nvme1n1): 16T used of 16T, 0 available, 100% - `/var/log` (nvme2n1): 49G used of 49G, 0 available, 100% - Root disk fine at 17% **What filled the data volume:** a stale 6.1T directory, `/var/opt/gitlab/postgresql/data18.bak.1783437348` (created about 2026-07-07 15:16 UTC, consistent with the July re-init of this node). Live data (about 9.5T) plus the stale copy (6.1T) exhausted the 16T volume. The failure is a capacity problem, not replication or timeline corruption: WAL restore was healthy up to the moment the disk filled. **Plan:** free the space (delete the stale directory, trim `/var/log`), restart Patroni and let PostgreSQL resume redo from its last restartpoint, fetching the missing WAL from the WAL-G archive. **A measured bail-out to a full rebuild applies** (see below), because the catch-up is large. **Criticality:** ~C3. The node does not serve production traffic and is not customer-facing. Similar earlier changes: [production#20574](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/20574) and [production#22846](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22846). **Reference:** {+link to dbo-issue-tracker issue / epic+} ### Verification and sizing results - **WAL step 1 (local `pg_wal`): PASS.** 5,209 contiguous segments from `0000000200172AEB000000AA` (redo start `172AEB/AAFCE9B0`) to `0000000200172B0000000002` (failing record `172B00/2006410`), no gaps. See [comment](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23452#note_4194563083). - **WAL step 2 (archive fetch): PASS.** `wal-fetch --turbo 0000000200172B0000000003` returned 16777216 bytes from `gs://gitlab-gprd-postgres-backup/pitr-walg-sec-v18`. See [comment](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23452#note_4194668671). - **WAL step 3 (full archive continuity listing from `172AEB` to the primary): SKIPPED by decision.** Accepted risk: a missing segment would only show up when replay reaches it, possibly hours in. Mitigation: monitor that replay keeps advancing (below). A stall at one segment means the archive is missing it, and the response is the rebuild fallback. It is not data corruption. - **WAL volume to replay:** primary at `0000000200173B89...` at 2026-11-09 16:42 UTC. From the redo start `172AEB/AA` that is about 4,254 log ids, roughly **17.4 TiB**. The primary generates about **43 MiB/s (2.5 GiB/min, about 3.6 TiB/day)**, which replay has to outrun. - **Estimated in-place catch-up time** (WAL to replay divided by replay rate minus 43 MiB/s): about 3.7 days at 100 MiB/s replay, about 2 days at 150 MiB/s, about 20 hours at 300 MiB/s. The rebuild fallback restores a recent disk snapshot of `patroni-sec-v18-102`, so only WAL written since the snapshot has to be replayed, not 17 TiB. Hence the measured bail-out below. ## Change Details 1. **Services Impacted** - ~"Service::PatroniSec" 1. **Change Technician** - @saadullah707 1. **Change Reviewer** - @bshah11 1. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-11-09 18:00 1. **Time tracking** - 48 hours 1. **Downtime Component** - none > [!note] > The archive node replays WAL with zero apply delay. The node has been down since 2026-11-04, so expect about 17 TiB of WAL to replay and a long catch-up. ## Preparation ### Change Reviewer checklist ~C4 ~C3: - [x] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ### Change Technician checklist - [x] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed. - [x] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [x] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access. - [ ] This Change Issue is linked to the appropriate Issue and/or Epic - [x] Change has been tested in staging and results noted in a comment on this issue. (N/A if DR nodes are not run in staging, same rationale as #20574.) - [x] A dry-run has been conducted and results noted in a comment on this issue. (N/A if not possible on a single production DR node.) - [ ] The change execution window respects the [Production Change Lock periods](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#production-change-lock-pcl). - [x] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` ## Detailed steps for the change ### Pre-execution steps - [x] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [x] The SRE on-call has been informed prior to the change. - [ ] Check for active ~severity::1 / ~severity::2 incidents. If one overlaps with this change's blast radius, reschedule. - [ ] **Confirm the rebuild fallback is viable:** a recent data-disk snapshot of `patroni-sec-v18-102` exists (`gcloud compute snapshots list --filter="sourceDisk~patroni-sec-v18-102-db-gprd-data" --sort-by=creationTimestamp --format=json | jq -r '.[-1]'`). Record the newest snapshot name and creation time: {+name / time+} ### Change steps - steps to take to execute the change *Estimated Time to Complete (mins)* - {+estimate; delete can take a while, catch-up could take 1 to 4 days+} - [x] Set label ~"change::in-progress" `/label ~change::in-progress` - [x] **Alert silence.** Create the silence before starting the change, at https://alerts.gitlab.net/#/silences (or via the [alert-management tool](https://gitlab.com/gitlab-com/gl-infra/db-migration/-/tree/master/alert-management)). - **Matchers:** - `env="gprd"` - `fqdn=~"postgres-sec-dr-archive-v18-01-db-gprd.*"` (regex match, so the full internal FQDN suffix does not need to be spelled out) - **Duration:** the whole change, including the catch-up (1 to 4 days), plus a margin. Extend it if the rebuild fallback is used. - **Comment:** link to this issue, `https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23452`. - [x] Start a tmux session so the work survives a dropped SSH connection: `tmux new -s cleanup` - [x] Record current state as a comment: `df -h`, `sudo gitlab-patronictl list`, `pg_controldata /var/opt/gitlab/postgresql/data18` (cluster state and latest checkpoint / redo location, expected redo start `172AEB/AAFCE9B0`). - [x] Attach the FATAL excerpt and the tail of `/var/log/gitlab/patroni/patroni.log` to this issue before truncating anything. - [x] Disable Chef so it cannot restart Patroni: `chef-client-disable "Recovering sec DR archive, production CR 23452"` - [x] Stop Patroni: `systemctl stop patroni` - [x] Confirm Patroni is stopped (expect `inactive`): `systemctl is-active patroni` - [x] Confirm no PostgreSQL processes (expect no output): `pgrep -a postgres` - [x] **Verify before deleting:** the live data directory is `data18` and not the `.bak` directory. - `readlink -f /var/opt/gitlab/postgresql/data` - `grep data_dir /var/opt/gitlab/patroni/patroni.yml` - `ls -ld /var/opt/gitlab/postgresql/data18*` - `lsof +D /var/opt/gitlab/postgresql/data18.bak.1783437348` (expect no output) - [x] Delete the stale directory: `ionice -c3 rm -rf /var/opt/gitlab/postgresql/data18.bak.1783437348` - [x] Check data disk space (expect about 9.5T used and about 6T free): `df -h /var/opt/gitlab` - [x] Find what filled `/var/log` (the top-level `du` hides `/var/log/gitlab`): `du -xh --max-depth=3 /var/log | sort -h | tail -15` and `journalctl --disk-usage`. Record findings in a comment: {+findings+} - [x] Free log space: delete rotated logs (`*.gz`, `*.1`) first; truncate only what is still large, e.g. `truncate -s 0 /var/log/gitlab/patroni/patroni.log`. Do **not** truncate `postgresql.csv`, it is needed to watch recovery. - [x] If the journal is large: `journalctl --vacuum-size=2G`. If syslog is large: `truncate -s 0 /var/log/syslog`, then `systemctl kill -s HUP rsyslog`. - [x] Check log disk space (expect well under 50%): `df -h /var/log` - [x] **Clean the interrupted prefetch:** a partial `0000000200172B0000000020` (4,685,824 bytes, cut off when the disk filled) and a `running` directory are in the prefetch cache. Remove them (prefetch is only a cache): - `ls -la /var/opt/gitlab/postgresql/data18/pg_wal/.wal-g/prefetch/running` - `rm -f /var/opt/gitlab/postgresql/data18/pg_wal/.wal-g/prefetch/0000000200172B0000000020` - `rm -f /var/opt/gitlab/postgresql/data18/pg_wal/.wal-g/prefetch/running/*` - [x] Start Patroni: `systemctl start patroni` - [x] In a second tmux pane, watch free space during catch-up: `watch -n 60 df -h /var/opt/gitlab /var/log` - [x] Watch recovery: `tail -f /var/log/gitlab/postgresql/postgresql.csv | grep -E 'redo starts|consistent recovery|restored log file|FATAL|PANIC'`. Expect `redo starts at 172AEB/AAFCE9B0`, then `consistent recovery state reached`, then a steady stream of `restored log file`. - [x] Check cluster state (expect `Standby Leader`, `in archive recovery` or `running`): `gitlab-patronictl list` - [ ] **Measured bail-out.** The first 10-20 minutes replay about 83 GB from local `pg_wal`, which is faster than archive-fed replay, so do not decide on that. After replay passes `172B00/02` (the node starts fetching from the archive), sample `gitlab-psql -Atc "select pg_last_wal_replay_lsn()"` twice, 10 minutes apart, and compute the replay rate. Net catch-up rate = replay rate minus about 43 MiB/s (primary generation). ETA = remaining WAL (primary LSN minus replay LSN) divided by the net rate. Record rate and ETA in a comment: {+rate / ETA+} - [ ] If ETA is more than about 24 hours, or clearly longer than the rebuild estimate, execute the **rebuild fallback** in the [Rollback](#rollback) section. - [ ] **Stall check (replaces the skipped archive gap check).** Every 30-60 minutes confirm the replay LSN keeps advancing: `gitlab-psql -c "select pg_is_in_recovery(), pg_last_wal_replay_lsn(), now() - pg_last_xact_replay_timestamp() as replay_lag;"`. If the LSN does not advance for more than 10 minutes, check the startup process: `ps -ef | grep 'startup' | grep -v grep` (it shows the segment it is waiting for) and test that segment with `wal-fetch` (see the note below). If the archive is missing it, execute the rebuild fallback. - [ ] Once replay lag is close to normal, re-enable Chef and run it: `chef-client-enable` then `chef-client` - [ ] Confirm `pg_replication_lag{env="gprd", fqdn=~"postgres-sec-dr-archive.*"}` returns to ~0 and stays there. Attach a before/after graph. - [ ] Confirm disk usage is stable and has headroom: `df -h /var/opt/gitlab /var/log`. - [ ] Expire the silence created in the alert silence step above once the node is caught up. - [ ] Follow-ups (open issues and link here): why `data18.bak.*` was left on the volume after the July re-init; why `/var/log` filled / log rotation; disk-space alerting for DR nodes; data growth versus the 16T volume size (live data about 9.5T); whether replay speed on this node needs tuning. {+links+} - [ ] Set label ~"change::complete" `/label ~change::complete` > [!note] wal-g command notes for this host > - `envdir` is `/usr/bin/envdir`, wal-g is `/opt/wal-g/bin/wal-g`; the node's own restore command is `/usr/bin/envdir /etc/wal-g.d/env /opt/wal-g/bin/wal-g wal-fetch --turbo %f %p`. > - Before `sudo -u gitlab-psql envdir ...`, `cd` into a directory `gitlab-psql` can read (not `/root`), otherwise `envdir` fails with `unable to switch to current directory: access denied`. Use a scratch dir such as `/tmp/walcheck` (owned by `gitlab-psql`, removed afterwards) as the destination. > - Spot check example: `cd /tmp/walcheck; sudo -u gitlab-psql /usr/bin/envdir /etc/wal-g.d/env /opt/wal-g/bin/wal-g wal-fetch --turbo <SEGMENT> /tmp/walcheck/seg; echo "exit=$?"; ls -l /tmp/walcheck/seg` ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change *Estimated Time to Complete (mins)* - {+estimate; a few hours expected: new disk from snapshot plus replay of WAL written since the snapshot, not measured+} Deleting the stale `.bak` directory is not reversible, but it is not needed: the node is a replica, and the live data lives in `data18`. **When to roll back (rebuild):** Postgres fails again at startup for any reason other than disk space (checksum errors, `invalid page`, missing files, missing WAL in the archive), or the measured bail-out or stall check above trips. **Approach: restoring with a disk snapshot.** This follows [Restoring with a disk-snapshot](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/postgres-dr-delayed/postgres-dr-replicas.md#restoring-with-a-disk-snapshot) from the DR replica runbook, in the same order. It is faster than a base-backup: a new disk is created from the latest data-disk snapshot of `patroni-sec-v18-102` (a Patroni host in the sec cluster, so the "snapshot from a Patroni host" branch of the runbook applies), mounted in place of the existing data disk, and then only the WAL written since the snapshot is fetched and replayed, not about 17 TiB. **Disks of this node** (`gcloud --project gitlab-production compute disks list --filter="name~postgres-sec-dr-archive"`, checked 2026-11-09), all `hyperdisk-balanced` in `us-east1-c`: | Disk | Size | Role | | --- | --- | --- | | `postgres-sec-dr-archive-v18-01-db-gprd` | 100 GB | boot disk, **not touched** | | `postgres-sec-dr-archive-v18-01-db-gprd-data` | 16000 GB | data disk (`/var/opt/gitlab`), **the only disk replaced** | | `postgres-sec-dr-archive-v18-01-db-gprd-log` | 50 GB | `/var/log`, **not touched** | Differences from the runbook, because it predates this setup: | Runbook | This node | | --- | --- | | `--type pd-ssd` | C4 machine type: **Hyperdisk** (`hyperdisk-balanced`). Reuse the type, IOPS and throughput of the current disk. | | Disk size not set (inherits from the snapshot) | Data disk is **16000 GB** and the sec cluster data disk is the same size, so the new disk inherits 16000 GB. The snapshot source disk size is checked in step 2. | | `gitlab-ctl stop` / `reconfigure` / `restart postgresql` | **Patroni**: `systemctl` and `gitlab-patronictl`. **Chef stays disabled until the node is `Standby Leader`.** | | `mv data* data`, `rm -rf pgbouncer patroni` | The data directory stays **`data18`** (Patroni `data_dir`). Remove only `pgbouncer`, and **restore this node's own Patroni config** instead of deleting it. | | `tf plan -target module.postgres-dr-archive -target module.postgres-dr-delayed` | Module names differ, see the Terraform check below. | **1. Stop Patroni and get the config of the current data disk** - [x] Stop Patroni and keep it from restarting: `sudo systemctl stop patroni ; sudo systemctl disable patroni`. Confirm Chef is still disabled (`chef-client-disable` from the change steps). - [x] Get the instance, zone, disk name, device name and labels. The data disk name and zone were confirmed in the disk table above (`postgres-sec-dr-archive-v18-01-db-gprd-data`, `us-east1-c`). The script uses an exact name match, so the boot and log disks are not picked up: ```sh set -x env="gprd" project="gitlab-production" instance="postgres-sec-dr-archive-v18-01-db-gprd" disk_name="${instance}-data" zone=$(gcloud --project $project compute disks list --filter="name=$disk_name" --format=json | jq -r 'first | .zone | split("/") | last') labels=$(gcloud --project $project compute disks list --filter="name=$disk_name" --format=json | jq -r 'first | .labels | to_entries | map("\(.key)=\(.value)") | join(",")') pd_name=$(gcloud --project $project compute instances describe $instance --zone $zone --format=json | jq -r --arg d "$disk_name" '.disks[] | select(.source | endswith("/" + $d)) | .deviceName') set +x echo "disk_name=$disk_name zone=$zone pd_name=$pd_name labels=$labels" ``` Expect `zone=us-east1-c`. `pd_name` is the device name the disk is attached with. The runbook assumes `persistent-disk-1`, so use the value read here instead. All values must be non-empty before continuing. - [x] Record the current disk settings. They are reused for the new disk so that the Terraform plan stays clean: ```sh gcloud --project $project compute disks describe $disk_name --zone $zone --format="yaml(type,sizeGb,provisionedIops,provisionedThroughput)" disk_type=$(gcloud --project $project compute disks describe $disk_name --zone $zone --format="value(type.basename())") iops=$(gcloud --project $project compute disks describe $disk_name --zone $zone --format="value(provisionedIops)") throughput=$(gcloud --project $project compute disks describe $disk_name --zone $zone --format="value(provisionedThroughput)") ``` Record: type `hyperdisk-balanced`, size `16000 GB` (both confirmed in the disk table above), IOPS / throughput: {+IOPS / throughput+}. **2. Select the snapshot (snapshot from a Patroni host)** - [x] Get the snapshot list for the data disk of the designated backup replica of the sec cluster, `patroni-sec-v18-102`, and take the most recent one: ```sh gcloud --project $project compute snapshots list --filter="sourceDisk~patroni-sec-v18-102-db-gprd-data" --sort-by=creationTimestamp --format=json | jq -r '.[-1]' snapshot_name=<name of most recent snapshot from the previous command> ``` Record snapshot name and creation time: {+name / time+}. The older the snapshot, the more WAL has to be replayed. - [x] Check the size of the snapshot's source disk. The new disk inherits it, so it must be **16000 GB**, the same as the current data disk, or the Terraform plan will show a size diff: `gcloud --project $project compute disks list --filter="name=patroni-sec-v18-102-db-gprd-data" --format="table(name,zone.basename(),sizeGb,type.basename())"`. Result: {+size+} - [x] If the most recent snapshot is not recent enough, take a new snapshot (confirm that `patroni-sec-v18-102` is the designated backup replica first, see the [snapshots runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/gcs-snapshots.md)): 1. SSH to `patroni-sec-v18-102`. 1. Run `/usr/local/bin/gcs-snapshot.sh` as the user `gitlab-psql`. Wait for the snapshot to finish. 1. List the snapshots again with the command above and set `snapshot_name` to the newest one. **3. Prepare this node** - [x] **Back up this node's Patroni config to the root disk** before the data disk is removed. The snapshot carries `patroni-sec-v18-102`'s config, which must not be used on this node: `sudo cp -a /var/opt/gitlab/patroni /root/patroni.cr-23452.bak && ls -l /root/patroni.cr-23452.bak/patroni.yml` - [x] On this node, adjust the `/etc/fstab` entry for the data disk to also work for the new disk (runbook step). The disk is attached with the device name `$pd_name` read above: ```sh sudo cp /etc/fstab /etc/fstab.cr-23452.bak pd_path="/dev/disk/by-id/google-${pd_name}" if test -L $pd_path; then sudo awk -i inplace 'match($0, /\/var\/opt\/gitlab/) { $0 = gensub(/^UUID=[^\ ]+/, "'$pd_path'", 1) } 1 {print $0}' /etc/fstab echo "Updated /etc/fstab" else echo "ERROR - double-check the persistent disk path as '${pd_path}' does not exist" fi grep /var/opt/gitlab /etc/fstab ``` The `grep` must show the `/dev/disk/by-id/google-...` path for `/var/opt/gitlab`. On a C4 (NVMe) node, do not continue if the ERROR line is printed. **4. Exchange the data disk with a new one created from the snapshot** (the disk keeps its name, so Terraform sees no diff) - [x] Run the following from a workstation with the variables set above. `--size` is not passed, so the disk inherits the size of the snapshot (16000 GB). Leave out `--provisioned-iops` or `--provisioned-throughput` if the recorded value is empty: ```sh # stop the instance gcloud --project $project compute instances stop $instance --zone $zone # detach the disk gcloud --project $project compute instances detach-disk $instance --disk $disk_name --zone $zone # delete the disk gcloud --project $project compute disks delete $disk_name --zone $zone # create new disk from snapshot (takes some time...) gcloud --project $project compute disks create $disk_name --type "$disk_type" --provisioned-iops "$iops" --provisioned-throughput "$throughput" --source-snapshot $snapshot_name --labels="$labels" --zone $zone # attach disk gcloud --project $project compute instances attach-disk $instance --disk $disk_name --device-name $pd_name --zone $zone # start instance gcloud --project $project compute instances start $instance --zone $zone ``` - [x] Confirm the new disk has the recorded type and size: `gcloud --project $project compute disks describe $disk_name --zone $zone --format="yaml(type,sizeGb,provisionedIops,provisionedThroughput)"` **5. On the node: prepare the snapshot data and start Patroni** (the snapshot is from a Patroni host) - [x] Ensure Postgres and Patroni are not running, and Chef is still disabled: `systemctl is-active patroni` (expect `inactive`), `pgrep -a postgres` (expect no output), and confirm the volume is mounted: `df -h /var/opt/gitlab`. - [x] Check the data directory and remove the dirs you do not need. Do **not** rename `data18` to `data`; Patroni expects `data18` here (`data_dir` in this node's `patroni.yml`): ```sh ls -ld /var/opt/gitlab/postgresql/data* cd /var/opt/gitlab sudo rm -rf pgbouncer sudo rm -rf patroni && sudo cp -a /root/patroni.cr-23452.bak patroni grep data_dir /var/opt/gitlab/patroni/patroni.yml ``` The restored `patroni.yml` must be this node's (`postgres-sec-dr-archive-v18-01-db-gprd`, cluster `gprd-patroni-sec-archive-v18`), and its `data_dir` must match the directory in `ls`. - [x] Fix permissions: ```sh cd /var/opt/gitlab sudo chown -R gitlab-psql:gitlab-psql postgresql sudo chgrp root postgresql ``` - [x] Check that the WAL needed from the snapshot's redo point is in the archive: `pg_controldata /var/opt/gitlab/postgresql/data18 | grep -E 'REDO WAL file|Latest checkpoint location'`, then spot check that segment with `wal-fetch` (see the wal-g note above). If it is missing, take a newer snapshot (step 2) and repeat from step 4. - [x] Start Patroni (this replaces `gitlab-ctl reconfigure postgresql`): `sudo systemctl enable patroni && sudo systemctl start patroni` - [x] Keep an eye on the CSV log for recovery and a `restartpoint complete` message: `tail -f /var/log/gitlab/postgresql/postgresql.csv | grep -E 'redo starts|consistent recovery|restored log file|restartpoint complete|FATAL|PANIC'`. It can take minutes to hours depending on how many WAL segments have to be replayed. If Patroni reports a system identifier mismatch against the cluster key `service/gprd-patroni-sec-archive-v18`, stop and ask the DBRE before touching the consul key. - [x] Wait until `sudo gitlab-patronictl list` shows the node as `Standby Leader` (running). **Keep Chef disabled until then.** - [ ] Re-enable and run Chef (this replaces the second `gitlab-ctl reconfigure postgresql`): `chef-client-enable` then `chef-client`. It should run quickly and finish without errors. - [x] Restart Postgres for good measure only if needed (for example a new Postgres version was installed by Chef, or `gitlab-patronictl list` shows `Pending restart`). Restart through Patroni, not `gitlab-ctl`: `sudo gitlab-patronictl restart gprd-patroni-sec-archive-v18 postgres-sec-dr-archive-v18-01-db-gprd` (verify the cluster and member names in `gitlab-patronictl list` first). - [x] Confirm `up{fqdn=~"postgres-sec-dr-archive.*"}` is 1 for node_exporter and postgres_exporter, and that `pg_replication_lag{env="gprd", fqdn=~"postgres-sec-dr-archive.*"}` goes to ~0. **6. Check there is no Terraform plan diff for the archive replica** - [x] Run `tf plan` in the matching environment (gprd), targeting the sec archive module. Verify the module name in config-mgmt `environments/database-gprd/` first, the runbook's `-target module.postgres-dr-archive -target module.postgres-dr-delayed` is outdated: `tf plan -out plan -target module.{+TBD module name+}` - [x] If there is a plan diff for mutable things like labels, apply it. If there is a plan diff for more severe things like the disk name, type or size, the new disk does not match the Terraform definition: you might have made a mistake and will have to repeat this procedure, or fix the definition in config-mgmt. - [ ] **Node deploy MR:** {+TBD+} - [ ] **Node re-build MR:** {+TBD+} (needed only if the VM or disks must be recreated through Terraform, with a data disk sized above current data plus headroom) **7. Close out** - [ ] Keep the silence in place until the node is caught up, then expire it (the silence created in the alert silence step of the change steps). - [ ] If the change is abandoned instead of completed through the rebuild, return the node to a stopped state and reschedule, and set label ~"change::aborted" `/label ~change::aborted`. ## Monitoring ### Key metrics to observe - Metric: `pg_replication_lag{env="gprd", fqdn=~"postgres-sec-dr-archive.*"}` - Location: Mimir / Grafana Explore (datasource mimir-gitlab-gprd); [replication lag dashboard](https://dashboards.gitlab.net/goto/bf11oczn494owf?orgId=1) - What changes should prompt a rollback or redo: lag does not fall after the start, or drops to zero then rises again (timeline switch). - Metric: replay LSN progress (`pg_last_wal_replay_lsn()`) versus the primary's current LSN - Location: on-host - What changes should prompt a rollback or redo: no advance for more than 10 minutes, or an ETA longer than the bail-out threshold. - Metric: disk free space on `/var/opt/gitlab` and `/var/log` for the archive host - Location: Grafana Explore / node dashboards, or `df -h` on the host - What changes should prompt a rollback or redo: free space trending toward zero during catch-up. - Metric: `gitlab-patronictl list` state and `postgresql.csv` recovery lines - Location: on-host - What changes should prompt a rollback or redo: any `FATAL` or `PANIC` other than a transient connection error, or no `redo starts` / `restored log file` progress.