<!-- Please review https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/ for the most recent information on our change plans and execution policies. --> # Production Change ## Change Summary Replace all nodes of the GPRD Patroni CI cluster (`gprd-patroni-ci-v17`) with nodes that have **right-sized 40 TiB Hyperdisk Balanced data disks** (down from 64 TiB). New nodes are built from a fresh snapshot behind the maintenance role, traffic is moved with a zero-downtime Patroni leader switchover, and the old nodes are decommissioned afterwards. Compute, PostgreSQL parameters, IOPS and throughput are **unchanged** (same `c4-highmem-144` / `c4-standard-48` machine types and the same chef roles), so no parameter-retune CR is needed for this cluster. The **LVM-backed backup node `patroni-ci-v17-01`** (4 × striped Hyperdisk PVs) is decommissioned and **not replaced**. One non-LVM backup node is kept. An LVM backup node can be re-added later via wal-g restore + weekend replay if LVM snapshots are ever needed again. **Reference issue:** https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/689 **Sizing rationale** (~30 TiB used, ~0.75 TiB/month net growth, 6-month buffer, 85%/90% SLO): https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/689#note_3741430037 **Prior similar CRs:** GSTG Registry CR-2 #22374, GPRD Sec #20372, GSTG Sec #20371. | Parameter | Current | New | |-----------|---------|-----| | Cluster nodes (leader + replicas) | 9 × `c4-highmem-144`, nodes `21`–`29` | 9 × `c4-highmem-144`, nodes `31`–`39` | | Backup nodes | `01` (LVM, `c4-standard-48`) + `20` (non-LVM, `c4-standard-48`) | `30` (non-LVM, `c4-standard-48`) only | | Data disk per node | 64 TiB Hyperdisk Balanced (~62.3 TiB fs, ~30 TiB used, 48%) | **40 TiB (40960 GiB)** Hyperdisk Balanced | | Data disk IOPS / throughput | 160,000 / 2,400 MiB/s | unchanged | | Log / OS disk | 250 GiB / 100 GiB Hyperdisk Balanced | unchanged | | PostgreSQL | 17, `data17`, same roles | unchanged | **Out of scope** (separate CRs): `postgres-ci-dr-archive-v17-01`, `postgres-ci-dr-delayed-v17-01`, the data-analytics replica and DBLab (all still 64 TiB, tracked in #689). The PG18 performance-test primary `patroni-ci-v18-105` is a separate cluster and is not touched. **Expected savings:** data-disk capacity drops from 11 × 64 TiB (704 TiB) to 10 × 40 TiB (400 TiB), a reduction of 304 TiB of Hyperdisk Balanced storage. Decommissioning LVM backup node `01` without replacement also removes one `c4-standard-48` VM, its 250 GiB log and 100 GiB OS disks, and the provisioned IOPS/throughput on its 4 LVM disks. Final figure at contracted rates to be confirmed with FinOps (same approach as #751). **Prerequisite:** pg_repack CR #22413 must be `change::complete` before this change is scheduled. ## Change Details <!-- To automatically add your change to the GitLab Production calendar update the following fields: - Time tracking - Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM) Bot: https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler --> 1. **Services Impacted** - ~"Service::PatroniCI" 1. **Change Technician** - @saadullah707 @vporalla 1. **Change Reviewer** - @pmistry2 @bshah11 1. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - **TBD** — to be set once pg_repack CR #22413 is `change::complete`, outside PCL periods 1. **Time tracking** - ~5–7 elapsed days (seed first node → snapshot → build 9 nodes → soak → switchover → 1-day observation → decommission); ~8 h hands-on 1. **Downtime Component** - None — zero downtime (new nodes built behind the maintenance role, zero-downtime Patroni leader switchover) > [!NOTE] > To find a **Change Reviewer**, use the [Reviewer Roulette dashboard](https://gitlab-org.gitlab.io/gitlab-roulette/?currentProject=production&order=1) > scoped to the `production` project. Spin the wheel. If roulette selects the current EOC (check > [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW)), > please re-spin to select another reviewer. > [!IMPORTANT] > If your change involves scheduled maintenance, add a step to set and > [unset maintenance mode](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/set_maintenance_window.md) > per our runbooks. This will make sure SLA calculations adjust for the maintenance period. ## Preparation > [!NOTE] > The following checklists must be done in advance, before setting the label ~"change::scheduled" ### Change Reviewer checklist <!-- To be filled out by the reviewer. --> ~C4 ~C3 ~C2 ~C1: - [ ] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [ ] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~"blocks deployments" and/or ~"blocks feature-flags" are applied as necessary. ### Change Technician checklist - [ ] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed. - [ ] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [ ] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access. - [ ] This Change Issue is linked to the appropriate Issue and/or Epic (dbo-issue-tracker#689, epic dbo&61) - [ ] Change has been tested in staging / `db-benchmarking` (empty-disk seed with module snapshot disabled → snapshot from the new node → build from the pinned snapshot → switchover) and results noted in a comment on this issue. - [ ] A dry-run has been conducted and results noted in a comment on this issue. - [ ] pg_repack CR #22413 is `change::complete` and no `repack.*` objects remain on `gitlab_partitions_dynamic.ci_builds`. - [ ] **MR-P (db-migration)** is **merged**: removes the stale hosts (`02`, `11`–`13`, `101`–`110`) from `dbre-toolkit/inventory/gprd-ci.yml` so the inventory matches the live cluster (`01`, `20`–`29`). This touches no infrastructure; it is only needed so the pre-change Ansible ping succeeds. - [ ] **MR1 (config-mgmt)** is prepared and reviewed but **not merged** (Atlantis plan attached to this issue as a comment, showing 1 instance + disks to add and **0 to destroy/replace**). - [ ] The change execution window respects the [Production Change Lock periods](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#production-change-lock-pcl). - [ ] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` - [ ] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). It is schedule to run every 2 hours. - [ ] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - [ ] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed [at least 2 weeks in advance](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#approval). Check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW) to find who will be on-call at the scheduled day and time. ## Detailed steps for the change ### How to read the steps - Every step says **where** to run the command: - **workstation** — your laptop with `chef-repo` checked out and `knife` configured. - **console** — `console-01-sv-gprd.c.gitlab-production.internal`, inside the `dbupgrade` tmux session. - **node NN** — logged in over SSH to `patroni-ci-v17-NN-db-gprd.c.gitlab-production.internal`. - Commands are run **one at a time**, in the order written. Do not skip ahead. - Every step has a **You should see** line. If you do not see it, stop and ask in the CR / `#g_database_operations` before continuing. - A **Gate** is a hard stop. Do not go past a gate until it is true. - Node names: full hostname = `patroni-ci-v17-NN-db-gprd.c.gitlab-production.internal`. Old nodes are `01`, `20`–`29`. New nodes are `30`–`39`. - Two helper commands are used throughout: - Cluster status (run on any patroni node): `sudo gitlab-patronictl list` - Which replicas receive read traffic (run on any patroni node or the console): `dig @127.0.0.1 -p 8600 ci-db-replica.service.consul. SRV +short | awk '{print $4}' | sort -u`. The raw `dig` output has one line per pgbouncer port (6432–6437) per node; the `awk | sort -u` keeps one hostname per node. When the same command is wrapped in `ssh host "..."`, the `$4` is written `\$4` so the local shell does not expand it. ### Merge-request and dependency map Every MR is merged at the point in the sequence where its inputs exist. Each MR (except MR-P and MR1) is raised live on the change day from an updated `main`, after the previous one has merged and its gate has passed. | MR | Repo | When | What it changes | Gate before merging | |----|------|------|-----------------|---------------------| | **MR-P** | db-migration | Preparation (before scheduling) | Remove stale hosts `02`, `11`–`13`, `101`–`110` from `dbre-toolkit/inventory/gprd-ci.yml` | None (no infra impact) | | **MR1** | config-mgmt | Day 1 | Set module-level `data_disk_snapshot = ""`; add node `30` (c4-standard-48, 40 TiB, maintenance role) | Atlantis plan: 1 add, **0 destroy/replace** | | **MR2** | config-mgmt | Day 2 | Add snapshot data source **pinned by name** to the `30` snapshot; set module-level `data_disk_snapshot` to it; give `30` the backup-replica role in TF | Snapshot from `30` is `READY`, 40960 GB | | **MR3** | config-mgmt | Day 2 | Add node `31` (c4-highmem-144, 40 TiB, maintenance role) | Plan shows disk restored from the `30` snapshot | | **MR4** | config-mgmt | Day 2 | Add nodes `32`–`39` (same spec) | Node `31` is `streaming` | | **MR5** | config-mgmt | Day 3 | Remove the maintenance role from `31`–`39` in TF | All nine new replicas serving reads; soak passed | | **MR6** | db-migration | Day 4 | Add nodes `30`–`39` to `dbre-toolkit/inventory/gprd-ci.yml` | All 21 hosts answer Ansible ping | | **MR7** | config-mgmt | Day 5 | Remove nodes `01`, `20`, `21`–`29`; remove the four `gcp_database_snapshot_gprd_ci_lvm_*` data sources and `lvm_data_disk_snapshots_map` | Plan destroys exactly 11 instances + their disks, nothing else | | **MR8** | config-mgmt | Day 5 | Module-level `data_disk_size` → 40960, drop per-node overrides, `data_disk_snapshot` back to the `most_recent` filter | Plan shows **no changes** to instances or disks | | **MR9** | db-migration | Day 5 | Remove `01`, `20`–`29` from the inventory (final list `30`–`39`) | — | Why the inventory is touched three times: Ansible `ping` fails on any host that does not resolve, so the stale entries must go before the first ping (MR-P); the new nodes can only be added once they exist (MR6); and the old nodes are removed after they are destroyed (MR9). ### Pre-execution steps > [!NOTE] > The following steps should be done right at the scheduled time of the change request. The [preparation steps](#preparation) are > listed below. - [ ] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [ ] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out and has provided approval with the ~eoc_approved label on the issue. - [ ] Check for [active ~severity::1 / ~severity::2 incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) before requesting approval. If one overlaps with this change's blast radius, reschedule. If it does not overlap, include a brief risk assessment in the `@sre-oncall` approval request; if the EOC is unavailable or does not approve, reschedule. - [ ] For ~C1, ~C2, or ~"blocks deployments" change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change. (Silences are created per phase in the steps below.) - [ ] **node 27 (current leader):** confirm no anti-wraparound / aggressive autovacuum is running on `ci_builds` (lesson from INC-13562): ```sh sudo gitlab-psql -c "SELECT pid, state, query_start, left(query, 120) FROM pg_stat_activity WHERE query ILIKE '%autovacuum%ci_builds%';" ``` **You should see:** `(0 rows)`. If a row is returned, wait for it to finish before starting Day 1 or Day 4. #### Pre-change setup *Estimated Time to Complete (mins)* - 60 - [ ] **workstation:** log in to the console server ```sh ssh -A console-01-sv-gprd.c.gitlab-production.internal ``` - [ ] **console:** put the `dbupgrade` private key in place. Copy it from 1Password → Production Vault → "db-upgrade user" into `~/.ssh/id_dbupgrade`, then: ```sh chmod 600 ~/.ssh/id_dbupgrade ln -s ~/.ssh/id_dbupgrade ~/.ssh/id_rsa ``` - [ ] **console:** open (or re-attach to) the tmux session used for the whole change ```sh sudo -u dbupgrade tmux a -t optimize || sudo -u dbupgrade tmux new -s optimize ``` - [ ] **console (tmux):** install Ansible and clone db-migration (this pulls the inventory with **MR-P** already merged) ```sh rm -rf ~/src/db-migration cd ~/src git clone https://gitlab.com/gitlab-com/gl-infra/db-migration.git cd db-migration git checkout master python3 -m venv ansible source ansible/bin/activate python3 -m pip install --upgrade pip python3 -m pip install ansible python3 -m pip install jmespath ansible --version ``` - [ ] **console (tmux):** confirm the toolkit reaches every current node ```sh cd ~/src/db-migration/dbre-toolkit ansible -i inventory/gprd-ci.yml patroni_cluster -m ping ``` **You should see:** 11 hosts (`01`, `20`–`29`) each with `"ping": "pong"`, and no `UNREACHABLE`. - [ ] **node 27:** record the current topology and paste the output as a comment on this issue ```sh sudo gitlab-patronictl list ``` **You should see:** leader `27`; replicas `21`–`26`, `28`, `29` all `streaming`; `01` and `20` `streaming` with tags `nofailover`, `noloadbalance`, `backup_node`. - [ ] :bar_chart: Capture the "before" baseline in Grafana Explore (`mimir-gitlab-gprd`, last 6h, `fqdn=~"patroni-ci-v17-(2[1-9]).*"`): cache hit ratio, blocks read, disk busy %, queue depth, iowait, IOPS, throughput, CPU %, page cache. Same Explore queries as #22374 with `env="gprd", type="patroni-ci"`. Paste screenshots/values as a comment. ### Change steps - steps to take to execute the change *Estimated Time to Complete (mins)* - ~480 hands-on across 5 days (plus unattended copy/snapshot time) #### Day 1 — Seed the first right-sized node (new backup node `30`) *Estimated Time* - 60 min hands-on, then 8–24 h unattended (~30 TiB copy) **Entry gate:** Pre-execution steps complete; Ansible ping OK; no Sev1/Sev2 overlapping; no anti-wraparound autovacuum on `ci_builds`. - [ ] On this issue: `/label ~change::in-progress` - [ ] Create a **48 h** silence at https://alerts.gitlab.net/#/silences with matchers `env="gprd"` and `fqdn=~"patroni-ci-v17-3.*"` (new nodes only; old nodes keep alerting). Paste the silence link as a comment. - [ ] Refresh and merge **MR1** (config-mgmt, `environments/database-gprd/patroni-ci.tf`). It does two things: 1. sets module-level `data_disk_snapshot = ""` so the new node gets an **empty** disk (the module has no per-node snapshot setting; existing disks are not touched) 2. adds node `30`: ```hcl 30 = { machine_type = var.machine_types["patroni-c4-standard-48"] min_cpu_platform = var.min_cpu_platforms["patroni-c4-cpu-platform"] data_disk_size = 40960 chef_run_list_extra = "\"role[${var.environment}-base-db-patroni-ci-v17-c4-std-48],role[${var.environment}-base-db-patroni-maintenance]\"" additional_labels = { shard = "backup" } zone = "us-east1-c" } ``` **Gate:** the Atlantis plan shows **1 instance + its disks to add** and **0 to destroy, 0 to replace**. Anything else → stop, do not apply, see "stale plan" in Rollback. - [ ] **workstation:** wait until the VM has bootstrapped and chef has run once (retry every few minutes) ```sh knife node show patroni-ci-v17-30-db-gprd.c.gitlab-production.internal -r ``` **You should see:** a run_list containing `role[gprd-base-db-patroni-ci-v17]`, `role[gprd-base-db-patroni-ci-v17-c4-std-48]` and `role[gprd-base-db-patroni-maintenance]`. - [ ] **node 30:** become root and start the copy from the cluster (same procedure as #22374 / #20372) ```sh sudo su - rm -rf /var/opt/gitlab/postgresql/data17/pg_wal/ systemctl stop patroni.service systemctl start patroni.service gitlab-patronictl reinit gprd-patroni-ci-v17 ``` Answer `y` when `patronictl` asks which member to reinit and to confirm. - [ ] **node 30:** check it started ```sh gitlab-patronictl list tail -n 50 /var/log/gitlab/postgresql/postgresql.csv ``` **You should see:** `patroni-ci-v17-30` listed as `Replica` with state `creating replica` (or `running`/`streaming` once the copy finishes), and no `FATAL` lines in the log. Note: if leader read load during the copy is a concern, seed with `pg_basebackup` from the backup replica `20` instead and then start patroni. - [ ] On this issue: `/label ~change::scheduled` - [ ] **Exit gate (several hours later), node 30:** ```sh sudo gitlab-patronictl list df -h /var/opt/gitlab ``` **You should see:** `30` as `Replica`, state `streaming`, `Lag in MB` ≈ 0, tags `nofailover`/`noloadbalance`; `df` shows a ~40T filesystem with ~30T used. Paste both outputs as a comment. Do not start Day 2 before this. #### Day 2 — Snapshot from `30`, build the 9 cluster nodes `31`–`39` *Estimated Time* - 4–6 h (snapshot of ~30 TiB + 9 disk restores) **Entry gate:** Day 1 exit gate met. - [ ] On this issue: `/label ~change::in-progress` - [ ] **node 20:** disable the old hourly snapshot cron so no new 64 TiB snapshots are created ```sh sudo su - gitlab-psql crontab -l crontab -e ``` In the editor put a `#` in front of the line containing `/usr/local/bin/gcs-snapshot.sh`, save and exit, then: ```sh crontab -l exit ``` **You should see:** the `gcs-snapshot.sh` line starting with `#`. - [ ] **node 01:** repeat exactly the same four commands as on node 20. **You should see:** the `gcs-snapshot.sh` line starting with `#`. - [ ] **workstation:** give node `30` the backup-replica role and remove the maintenance role ```sh knife node run_list add patroni-ci-v17-30-db-gprd.c.gitlab-production.internal "role[gprd-base-db-patroni-ci-backup-replica]" knife node run_list remove patroni-ci-v17-30-db-gprd.c.gitlab-production.internal "role[gprd-base-db-patroni-maintenance]" ssh patroni-ci-v17-30-db-gprd.c.gitlab-production.internal "sudo chef-client" ``` **You should see:** chef-client finishing with `Chef Infra Client finished` and no error. - [ ] **node 30:** confirm `30` is a backup node and is **not** receiving read traffic ```sh sudo gitlab-patronictl list dig @127.0.0.1 -p 8600 ci-db-replica.service.consul. SRV +short | awk '{print $4}' | sort -u ``` **You should see:** `30` with tags `backup_node`, `console_node`, `nofailover`, `noloadbalance`; the `dig` output lists only old replicas (`21`–`26`, `28`, `29`), not `30`. - [ ] **node 30:** take the first 40 TiB snapshot ```sh sudo su - gitlab-psql /usr/local/bin/gcs-snapshot.sh exit ``` - [ ] **workstation:** find the snapshot name and confirm it is ready (repeat until `READY`) ```sh gcloud compute snapshots list --project gitlab-production --filter="sourceDisk~patroni-ci-v17-30-db-gprd-data" --format="table(name,status,diskSizeGb,creationTimestamp,sourceDisk.basename())" ``` **Gate:** one row with `STATUS = READY` and `DISK_SIZE_GB = 40960`. Its `NAME` is **auto-generated** by `gcs-snapshot.sh` (12 random characters, e.g. `axrfl40zxy1u`; only the LVM snapshots carry the host name). Write the `NAME` down; it is used in MR2. - [ ] Raise and merge **MR2** (config-mgmt). It does three things: 1. adds a snapshot data source pinned to the name from the previous step: ```hcl data "google_compute_snapshot" "gcp_database_snapshot_gprd_ci_40t" { name = "<NAME from the previous step, e.g. axrfl40zxy1u>" project = "gitlab-production" } ``` 2. sets module-level `data_disk_snapshot = data.google_compute_snapshot.gcp_database_snapshot_gprd_ci_40t.id` 3. changes node `30`'s `chef_run_list_extra` to `"\"role[${var.environment}-base-db-patroni-ci-v17-c4-std-48],role[${var.environment}-base-db-patroni-ci-backup-replica]\""` (so Terraform matches what was done with knife) **Gate:** the plan shows metadata-only changes on `30` and **0 destroy, 0 replace**. - [ ] Raise and merge **MR3** (config-mgmt): add node `31` ```hcl 31 = { data_disk_size = 40960 chef_run_list_extra = "\"role[${var.environment}-base-db-patroni-ci-v17-c4-hm-144],role[${var.environment}-base-db-patroni-maintenance]\"" zone = "us-east1-d" } ``` (module defaults give `c4-highmem-144`, 160k IOPS, 2400 MiB/s.) **Gate:** the plan shows the new data disk created **from the `30` snapshot**, 1 add, 0 destroy. - [ ] **workstation:** wait for `31` to bootstrap (retry every few minutes) ```sh knife node show patroni-ci-v17-31-db-gprd.c.gitlab-production.internal -r ``` **You should see:** a run_list containing `role[gprd-base-db-patroni-maintenance]`. - [ ] **node 31:** start Patroni and check it joins the cluster ```sh sudo systemctl enable patroni.service sudo systemctl start patroni.service sudo gitlab-patronictl list sudo tail -n 50 /var/log/gitlab/postgresql/postgresql.csv ``` **Gate:** `31` is `Replica`, `streaming`, lag ≈ 0, tags `nofailover`/`noloadbalance`; no `FATAL`/`ERROR` in the log. Only then continue. - [ ] Raise and merge **MR4** (config-mgmt): add nodes `32`–`39` with the same block as `31`, zones alternating `us-east1-b`, `us-east1-c`, `us-east1-d`. **Gate:** the plan shows 8 adds from the `30` snapshot, 0 destroy. - [ ] **node 32:** start Patroni ```sh sudo systemctl enable patroni.service sudo systemctl start patroni.service sudo gitlab-patronictl list ``` **You should see:** `32` as `Replica`, `streaming` (may take a few minutes to leave `starting`). - [ ] **node 33:** same three commands. **You should see:** `33` `streaming`. - [ ] **node 34:** same three commands. **You should see:** `34` `streaming`. - [ ] **node 35:** same three commands. **You should see:** `35` `streaming`. - [ ] **node 36:** same three commands. **You should see:** `36` `streaming`. - [ ] **node 37:** same three commands. **You should see:** `37` `streaming`. - [ ] **node 38:** same three commands. **You should see:** `38` `streaming`. - [ ] **node 39:** same three commands. **You should see:** `39` `streaming`. - [ ] **workstation:** check the logs of all new nodes in one go ```sh knife ssh 'name:patroni-ci-v17-3*' 'sudo tail -n 50 /var/log/gitlab/postgresql/postgresql.csv | grep -E "FATAL|ERROR|huge" || echo OK' ``` **You should see:** `OK` for every node. - [ ] On this issue: `/label ~change::scheduled` - [ ] **Exit gate, node 30:** ```sh sudo gitlab-patronictl list dig @127.0.0.1 -p 8600 ci-db-replica.service.consul. SRV +short | awk '{print $4}' | sort -u ``` **You should see:** `30`–`39` all `streaming`, lag ≈ 0; `31`–`39` tagged `nofailover`/`noloadbalance`; `dig` still lists only the old replicas. Paste the output as a comment. #### Day 3 — Move read traffic to the new replicas *Estimated Time* - 3–4 h incl. soak **Entry gate:** Day 2 exit gate met; error ratios on the dashboard flat for the last 6 h. Dashboard for this whole day: https://dashboards.gitlab.net/d/patroni-ci-main/patroni-ci3a-overview?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd — watch **connections per node**, **query latency**, **service error ratio**. - [ ] On this issue: `/label ~change::in-progress` - [ ] Create a **6 h** silence at https://alerts.gitlab.net/#/silences with matchers `env="gprd"` and `fqdn=~"patroni-ci-v17.*"`. **Step 3a — put the new replicas into service, one at a time.** For each node: run the three commands, then the check, then look at the dashboard for ~5 minutes before moving to the next node. - [ ] **workstation, node 31:** ```sh knife node run_list remove patroni-ci-v17-31-db-gprd.c.gitlab-production.internal "role[gprd-base-db-patroni-maintenance]" ssh patroni-ci-v17-31-db-gprd.c.gitlab-production.internal "sudo chef-client" ssh patroni-ci-v17-31-db-gprd.c.gitlab-production.internal "dig @127.0.0.1 -p 8600 ci-db-replica.service.consul. SRV +short | awk '{print \$4}' | sort -u" ``` **You should see:** `patroni-ci-v17-31` now in the `dig` list; on the dashboard, connections appear on `31` within a few minutes and the error ratio stays flat. - [ ] **workstation, node 32:** same three commands with `32`. **You should see:** `32` added to the list, error ratio flat. - [ ] **workstation, node 33:** same with `33`. - [ ] **workstation, node 34:** same with `34`. - [ ] **workstation, node 35:** same with `35`. - [ ] **workstation, node 36:** same with `36`. - [ ] **workstation, node 37:** same with `37`. - [ ] **workstation, node 38:** same with `38`. - [ ] **workstation, node 39:** same with `39`. **Gate:** `dig` lists all of `21`–`26`, `28`, `29` **and** `31`–`39`; all nine new nodes show connections; error ratios flat. - [ ] Raise and merge **MR5** (config-mgmt): remove `role[${var.environment}-base-db-patroni-maintenance]` from the `chef_run_list_extra` of `31`–`39`, so Terraform matches what knife did. **Gate:** the plan shows metadata-only changes, 0 destroy. **Step 3b — soak.** - [ ] :bar_chart: Let the new replicas serve reads for **at least 1 hour**. Re-run the baseline Explore queries scoped to `fqdn=~"patroni-ci-v17-3[1-9].*"` and paste the results as a comment. **Gate / safe abort point:** cache hit ratio ≈ baseline, disk busy < 20%, queue depth < 1, iowait ≈ 0, no error-ratio rise. Old nodes are still serving, so aborting here is a plain revert (see Rollback, Day 3). **Step 3c — take the old replicas out of service, one at a time.** Same rhythm: three commands, check, watch the dashboard ~5 minutes, next node. - [ ] **workstation, node 21:** ```sh knife node run_list add patroni-ci-v17-21-db-gprd.c.gitlab-production.internal "role[gprd-base-db-patroni-maintenance]" ssh patroni-ci-v17-21-db-gprd.c.gitlab-production.internal "sudo chef-client" ssh patroni-ci-v17-21-db-gprd.c.gitlab-production.internal "dig @127.0.0.1 -p 8600 ci-db-replica.service.consul. SRV +short | awk '{print \$4}' | sort -u" ``` **You should see:** `patroni-ci-v17-21` gone from the `dig` list; its connections drain on the dashboard; error ratio flat. - [ ] **workstation, node 22:** same three commands with `22`. - [ ] **workstation, node 23:** same with `23`. - [ ] **workstation, node 24:** same with `24`. - [ ] **workstation, node 25:** same with `25`. - [ ] **workstation, node 26:** same with `26`. - [ ] **workstation, node 28:** same with `28`. - [ ] **workstation, node 29:** same with `29`. (Node `27` is the leader and is handled on Day 4. Nodes `01` and `20` are backup nodes and already out of the read pool.) - [ ] Expire the 6 h silence. - [ ] On this issue: `/label ~change::scheduled` - [ ] **Exit gate, node 30:** ```sh dig @127.0.0.1 -p 8600 ci-db-replica.service.consul. SRV +short | awk '{print $4}' | sort -u ``` **You should see:** only `patroni-ci-v17-31` … `patroni-ci-v17-39`. On the dashboard, connections are on `31`–`39` and leader `27` only; error ratios flat. **Wait one full day** before Day 4. #### Day 4 — Leader switchover *Estimated Time* - 60 min **Entry gate:** Day 3 exit gate met and 24 h of clean observation; no Sev1/Sev2 overlapping; Release Managers acknowledged today's window. - [ ] On this issue: `/label ~change::in-progress` - [ ] **node 27:** re-check that nothing heavy is running on `ci_builds` ```sh sudo gitlab-psql -c "SELECT pid, state, query_start, left(query, 120) FROM pg_stat_activity WHERE query ILIKE '%autovacuum%ci_builds%' OR query ILIKE '%repack%';" ``` **You should see:** `(0 rows)`. - [ ] Create a **4 h** silence at https://alerts.gitlab.net/#/silences with matchers `env="gprd"` and `fqdn=~"patroni-ci-v17.*"`. - [ ] Raise and merge **MR6** (db-migration): add these ten lines under `patroni_cluster: hosts:` in `dbre-toolkit/inventory/gprd-ci.yml`: ```yaml patroni-ci-v17-30-db-gprd.c.gitlab-production.internal: patroni-ci-v17-31-db-gprd.c.gitlab-production.internal: patroni-ci-v17-32-db-gprd.c.gitlab-production.internal: patroni-ci-v17-33-db-gprd.c.gitlab-production.internal: patroni-ci-v17-34-db-gprd.c.gitlab-production.internal: patroni-ci-v17-35-db-gprd.c.gitlab-production.internal: patroni-ci-v17-36-db-gprd.c.gitlab-production.internal: patroni-ci-v17-37-db-gprd.c.gitlab-production.internal: patroni-ci-v17-38-db-gprd.c.gitlab-production.internal: patroni-ci-v17-39-db-gprd.c.gitlab-production.internal: ``` - [ ] **console (tmux):** pull the inventory and confirm the toolkit reaches **all 21** nodes ```sh cd ~/src/db-migration git pull cd dbre-toolkit source ../ansible/bin/activate ansible -i inventory/gprd-ci.yml patroni_cluster -m ping ``` **Gate:** 21 hosts (`01`, `20`–`39`) each `"ping": "pong"`, none `UNREACHABLE`. The playbook must not run against a partial inventory. - [ ] **console (tmux):** run the zero-downtime switchover ```sh export PYTHONUNBUFFERED=1 cd ~/src/db-migration/dbre-toolkit ansible-playbook -i inventory/gprd-ci.yml switchover_patroni_leader.yml -e "non_interactive=false" 2>&1 | ts | tee -a ansible_switchover_patroni_leader_gprd-ci_$(date +%Y%m%d).log ``` When the playbook asks for the candidate, choose one of `patroni-ci-v17-31` … `39`. Confirm each prompt after reading it. **You should see:** the playbook end with `failed=0` for every host. - [ ] **node 30:** verify the new leader ```sh sudo gitlab-patronictl list ``` **Gate:** `Leader` is one of `31`–`39`; every other member is `streaming`; `27` is now a `Replica`. - [ ] Dashboard: confirm pgbouncer-ci and pgbouncer-sidekiq-ci moved to the new primary (connections appear on the new leader) and the `transactions_primary` error ratio is flat. - [ ] Dashboard: confirm `postgres-ci-dr-archive-v17-01` and `postgres-ci-dr-delayed-v17-01` replication lag is recovering (not growing) 10 minutes after the switchover. - [ ] **workstation:** take the old leader `27` out of the read pool ```sh knife node run_list add patroni-ci-v17-27-db-gprd.c.gitlab-production.internal "role[gprd-base-db-patroni-maintenance]" ssh patroni-ci-v17-27-db-gprd.c.gitlab-production.internal "sudo chef-client" ssh patroni-ci-v17-27-db-gprd.c.gitlab-production.internal "dig @127.0.0.1 -p 8600 ci-db-replica.service.consul. SRV +short | awk '{print \$4}' | sort -u" ``` **You should see:** `27` not in the `dig` list; only `31`–`39` minus the new leader. - [ ] Expire the 4 h silence. - [ ] On this issue: `/label ~change::scheduled` - [ ] **Exit gate:** all writes and reads on `31`–`39`; `01`, `20`–`29` idle behind maintenance; error ratios flat. **Wait one full day** before Day 5. This is the last point where the old nodes can be brought back without a rebuild. #### Day 5 — Decommission old nodes (`01`, `20`, `21`–`29`) *Estimated Time* - 90 min **Entry gate:** Day 4 exit gate met and 24 h of clean observation. - [ ] On this issue: `/label ~change::in-progress` - [ ] Create a **4 h** silence at https://alerts.gitlab.net/#/silences with matchers `env="gprd"` and `fqdn=~"patroni-ci-v17-(0|2).*"`. - [ ] **workstation:** confirm hourly snapshots are now coming from `30` ```sh gcloud compute snapshots list --project gitlab-production --filter="sourceDisk~patroni-ci-v17-30-db-gprd-data" --sort-by=~creationTimestamp --limit=3 --format="table(name,status,creationTimestamp)" ``` **Gate:** the newest snapshot is `READY` and less than 2 hours old. Do not remove `01`/`20` without it. - [ ] Raise and merge **MR7** (config-mgmt). It does two things: 1. removes nodes `01`, `20`, `21`, `22`, `23`, `24`, `25`, `26`, `27`, `28`, `29` from the `nodes` map 2. removes the four `data "google_compute_snapshot" "gcp_database_snapshot_gprd_ci_lvm_0..3"` blocks and the `lvm_data_disk_snapshots_map = { ... }` argument (no LVM node remains; those `most_recent` filters would fail once the LVM disks are gone) **Gate:** the plan destroys **exactly 11 instances and their disks and nothing else** (no change to `30`–`39`, the load balancer, backend service or subnets). If the plan shows unrelated deletes, rebase and re-plan. Never apply a stale plan. - [ ] **workstation:** remove the 11 old hosts from Chef, one at a time (each pair: node then client) ```sh knife node delete patroni-ci-v17-01-db-gprd.c.gitlab-production.internal -y knife client delete patroni-ci-v17-01-db-gprd.c.gitlab-production.internal -y knife node delete patroni-ci-v17-20-db-gprd.c.gitlab-production.internal -y knife client delete patroni-ci-v17-20-db-gprd.c.gitlab-production.internal -y knife node delete patroni-ci-v17-21-db-gprd.c.gitlab-production.internal -y knife client delete patroni-ci-v17-21-db-gprd.c.gitlab-production.internal -y knife node delete patroni-ci-v17-22-db-gprd.c.gitlab-production.internal -y knife client delete patroni-ci-v17-22-db-gprd.c.gitlab-production.internal -y knife node delete patroni-ci-v17-23-db-gprd.c.gitlab-production.internal -y knife client delete patroni-ci-v17-23-db-gprd.c.gitlab-production.internal -y knife node delete patroni-ci-v17-24-db-gprd.c.gitlab-production.internal -y knife client delete patroni-ci-v17-24-db-gprd.c.gitlab-production.internal -y knife node delete patroni-ci-v17-25-db-gprd.c.gitlab-production.internal -y knife client delete patroni-ci-v17-25-db-gprd.c.gitlab-production.internal -y knife node delete patroni-ci-v17-26-db-gprd.c.gitlab-production.internal -y knife client delete patroni-ci-v17-26-db-gprd.c.gitlab-production.internal -y knife node delete patroni-ci-v17-27-db-gprd.c.gitlab-production.internal -y knife client delete patroni-ci-v17-27-db-gprd.c.gitlab-production.internal -y knife node delete patroni-ci-v17-28-db-gprd.c.gitlab-production.internal -y knife client delete patroni-ci-v17-28-db-gprd.c.gitlab-production.internal -y knife node delete patroni-ci-v17-29-db-gprd.c.gitlab-production.internal -y knife client delete patroni-ci-v17-29-db-gprd.c.gitlab-production.internal -y ``` **You should see:** `Deleted node[...]` / `Deleted client[...]` for each. - [ ] **node 30:** confirm the cluster and Consul only know the new nodes ```sh sudo gitlab-patronictl list dig @127.0.0.1 -p 8600 ci-db-replica.service.consul. SRV +short | awk '{print $4}' | sort -u ``` **You should see:** only `30`–`39` in `patronictl list`; only `31`–`39` (minus the leader) in `dig`. - [ ] Raise and merge **MR8** (config-mgmt). It does three things: 1. sets `var.data_disk_sizes["patroni-ci-v17"]` to `40960` (module-level `data_disk_size`) 2. deletes the `data_disk_size = 40960` line from each of nodes `30`–`39` 3. points `data_disk_snapshot` back at `data.google_compute_snapshot.gcp_database_snapshot_gprd_ci.id` (the `most_recent` filter; only 40 TiB snapshots exist now) **Gate:** the plan shows **no changes** to instances or disks (no-op or metadata only). - [ ] Raise and merge **MR9** (db-migration): delete the `01`, `20`, `21`–`29` lines from `dbre-toolkit/inventory/gprd-ci.yml`, leaving `30`–`39`. - [ ] **console (tmux):** ```sh cd ~/src/db-migration git pull cd dbre-toolkit ansible -i inventory/gprd-ci.yml patroni_cluster -m ping ``` **You should see:** 10 hosts, all `pong`. - [ ] Expire the 4 h silence. - [ ] **node 30:** collect the final evidence and post it as a comment on this issue **and** on dbo-issue-tracker#689 ```sh sudo gitlab-patronictl list df -h /var/opt/gitlab ``` - [ ] On this issue: `/label ~change::complete` ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change *Estimated Time to Complete (mins)* - 120 Pick the section that matches how far you got. Commands are one per node, same shape as the forward steps. **Rollback on Day 1 or Day 2** (new nodes exist, none serve traffic) - [ ] Revert MR1–MR4 in config-mgmt (this destroys `30`–`39` and their disks). Check the plan destroys only `3x` nodes. - [ ] **node 20:** `sudo su - gitlab-psql`, `crontab -e`, remove the `#` in front of `gcs-snapshot.sh`, save, `crontab -l`, `exit`. - [ ] **node 01:** same as node 20. - [ ] On this issue: `/label ~change::aborted` **Rollback on Day 3** (reads moved, leader unchanged) - [ ] **workstation:** put each old replica back into service. Run for `21`, then check, then `22`, and so on for `23`, `24`, `25`, `26`, `28`, `29`: ```sh knife node run_list remove patroni-ci-v17-21-db-gprd.c.gitlab-production.internal "role[gprd-base-db-patroni-maintenance]" ssh patroni-ci-v17-21-db-gprd.c.gitlab-production.internal "sudo chef-client" ssh patroni-ci-v17-21-db-gprd.c.gitlab-production.internal "dig @127.0.0.1 -p 8600 ci-db-replica.service.consul. SRV +short | awk '{print \$4}' | sort -u" ``` **You should see:** the node back in the `dig` list after each pair. - [ ] **workstation:** take each new replica out of service. Run for `31`, check, then `32` … `39`: ```sh knife node run_list add patroni-ci-v17-31-db-gprd.c.gitlab-production.internal "role[gprd-base-db-patroni-maintenance]" ssh patroni-ci-v17-31-db-gprd.c.gitlab-production.internal "sudo chef-client" ssh patroni-ci-v17-31-db-gprd.c.gitlab-production.internal "dig @127.0.0.1 -p 8600 ci-db-replica.service.consul. SRV +short | awk '{print \$4}' | sort -u" ``` **You should see:** only `21`–`26`, `28`, `29` in the `dig` list at the end. - [ ] Revert MR1–MR5 in config-mgmt; re-enable the snapshot cron on `20` and `01` (as in the Day 1/2 rollback). - [ ] On this issue: `/label ~change::aborted` **Rollback on Day 4** (after the switchover, before decommission) - [ ] **workstation:** put the old replicas `21`–`26`, `28`, `29` back into service exactly as in the Day 3 rollback (`remove` maintenance, `chef-client`, `dig`, one node at a time). - [ ] **workstation:** also remove maintenance from the old leader `27`: ```sh knife node run_list remove patroni-ci-v17-27-db-gprd.c.gitlab-production.internal "role[gprd-base-db-patroni-maintenance]" ssh patroni-ci-v17-27-db-gprd.c.gitlab-production.internal "sudo chef-client" ``` - [ ] **workstation:** take the new replicas `31`–`39` out of service exactly as in the Day 3 rollback (`add` maintenance, one node at a time). Skip the current leader for now. - [ ] **console (tmux):** switch the leader back to one of `21`–`29` ```sh cd ~/src/db-migration/dbre-toolkit ansible-playbook -i inventory/gprd-ci.yml switchover_patroni_leader.yml -e "non_interactive=false" 2>&1 | ts | tee -a ansible_switchover_patroni_leader_gprd-ci_rollback_$(date +%Y%m%d).log ``` Choose an old node (`21`–`29`) as the candidate. - [ ] **workstation:** add the maintenance role to the former new leader (the `3x` node that was leader), same `add` + `chef-client` commands. - [ ] **node 30:** `sudo gitlab-patronictl list` — **You should see:** leader in `21`–`29`; dashboard shows pgbouncer on the old primary and flat error ratios. - [ ] Revert MR1–MR6; re-enable the snapshot cron on `20` and `01`. - [ ] On this issue: `/label ~change::aborted` **After Day 5 (decommission applied)** - [ ] The old nodes cannot be recovered. Recovery = rebuild from the `30` snapshots. The data is intact on the new cluster. This is why Day 4 → Day 5 has a full day of observation. - [ ] On this issue: `/label ~change::aborted` **Stale Atlantis plan:** if any plan shows deletes or changes you did not expect, do **not** apply. Rebase the MR on `main`, re-plan, and compare. Unexpected deletes usually mean someone else's applied-but-unmerged MR. ## Monitoring ### Key metrics to observe - Metric: Patroni CI overview — connections per node, query latency, saturation - Location: https://dashboards.gitlab.net/d/patroni-ci-main/patroni-ci3a-overview?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd - What changes to this metric should prompt a rollback: connections not moving onto the new nodes / off the old nodes as expected after each role change; sustained saturation on the new nodes (CPU, disk sustained write) - Metric: Service error ratio, `transactions_primary` and `transactions_replica` SLI error ratios - Location: same dashboard (SLI panels) - What changes to this metric should prompt a rollback: any sustained rise after adding/removing a node or after the switchover - Metric: pgbouncer-ci / pgbouncer-sidekiq-ci error ratio - Location: same dashboard (pgbouncer SLI panel) - What changes to this metric should prompt a rollback: sustained rise after the switchover (pgbouncer not following the new primary) - Metric: Replication lag — cluster replicas plus `postgres-ci-dr-archive-v17-01` and `postgres-ci-dr-delayed-v17-01` - Location: same dashboard (replication panels) - What changes to this metric should prompt a rollback: any new node failing to reach `streaming`, or lag not recovering after the switchover - Metric: Disk busy %, IOPS, throughput on the new 40 TiB disks (`nvme1n1`) - Location: host stats / Explore in `mimir-gitlab-gprd` scoped to `patroni-ci-v17-3[0-9].*` - What changes to this metric should prompt a rollback: sustained disk busy > 80% or throughput approaching the 2,400 MiB/s cap (IOPS/throughput are unchanged, so this is not expected) - Logs: Kibana `pubsub-postgres-inf-gprd*`, filter `json.hostname: patroni-ci-v17-3*` — FATAL / hugepage / shared-memory errors after each start