# Production Change ## Change Summary GSTG CR = https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22707 Follow-up to [INC-12794 / production-engineering#29507](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29507): during the incident an inactive replication slot retained ~4 TB of WAL and pushed the gprd `patroni-main` primary's disk to ~88%. We agreed with the Siphon team (@ahegyi, @arun.sori — [sign-off](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29507#note_3674723808)) to lower the per-slot WAL retention cap (`max_slot_wal_keep_size`) from ~10% to ≤7% of the data disk. This is the production step: **gprd `patroni-main-v17`, `5200GB` → `3640GB`**. The identical procedure was executed successfully on gstg `patroni-main-v18` on 2026-08-18 (gitlab-com/gl-infra/production#22707, `change::complete` — chef + reload, no restart, all slots stayed `reserved`), after a rehearsal on a db-benchmarking sandbox cluster. `max_slot_wal_keep_size` is a **reload-only** parameter — no restart, no downtime. **Siphon consideration:** at a 3640GB cap, the Siphon logical slot would be invalidated if its consumer stalled for roughly ~15h at average WAL rates (~7h at peak) — numbers from [Rafael's analysis in 29507](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29507#note_3655803043). Current Siphon retained WAL baseline (2026-08-18, via Mimir): **~0.6GB**. The safety pre-check below gates execution, and `SiphonLogicalReplicationSlotLagHigh` (pages at >100GiB lag) intentionally stays live during the change. MR: gitlab-com/gl-infra/chef-repo!7829 ## Change Details 1. **Services Impacted** - ~"Service::Patroni" 2. **Change Technician** - @pmistry2 3. **Change Reviewer** - @rhenchen.gitlab 4. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-08-24 21:30 (Monday 2:30 PM Pacific; flexible — reload-only. Rescheduled from 2026-08-20 due to active Sev2 INC-13334) 5. **Time tracking** - 120 6. **Downtime Component** - no downtime ## Preparation > [!note] > > The following checklists must be done in advance, before setting the label ~"change::scheduled" ### Change Reviewer checklist ~C4 ~C3 ~C2 ~C1: - [x] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [x] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~"blocks deployments" and/or ~"blocks feature-flags" are applied as necessary. ### Change Technician checklist - [x] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed. - [x] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [x] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access. - [x] This Change Issue is linked to the appropriate Issue and/or Epic - [x] Change has been tested in staging and results noted in a comment on this issue: executed on gstg `patroni-main-v18` as gitlab-com/gl-infra/production#22707 (`change::complete`, 2026-08-18), plus sandbox rehearsal evidence in that CR's comments. - [x] A dry-run has been conducted and results noted in a comment on this issue. - [x] The change execution window respects the [Production Change Lock periods](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#production-change-lock-pcl). - [x] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` - [x] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). It is schedule to run every 2 hours. - [x] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - [x] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [x] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed [at least 2 weeks in advance](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#approval). Check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW) to find who will be on-call at the scheduled day and time. ## Detailed steps for the change ### Pre-execution steps - [x] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [x] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out. - [x] The SRE on-call provided approval with the ~eoc_approved label on the issue. - [x] For ~C1, ~C2, or ~"blocks deployments" change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [x] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) that are ~severity::1 or ~severity::2 - [x] **Alert silences: none planned** — confirm this still holds at execution time. > [!note] > Reload-only change with no alert-generating events: no alert rules key on `pending_restart`, and the Siphon slot alert (`SiphonLogicalReplicationSlotLagHigh`) is intentionally left **live** — it is the safety signal for this change, not noise to silence. - [x] **Confirm cluster is healthy** (on any main cluster member): ```shell sudo gitlab-patronictl list ``` Expected topology (verified 2026-08-18 via Mimir): leader `patroni-main-v17-01`, 11 streaming replicas (`02`–`11`, `102`). `102` is the backup/LVM-snapshot node — expect `nofailover`/`noloadbalance` tags. - [x] **Safety pre-check** — run on the leader (currently `patroni-main-v17-01-db-gprd.c.gitlab-production.internal`): ```sql SELECT slot_name, slot_type, active, wal_status, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal, pg_size_pretty(safe_wal_size) AS safe_wal_size FROM pg_replication_slots; ``` **Gate:** no slot may be retaining between **3640GB and 5200GB** of WAL — such a slot would be invalidated by the new cap at reload. All slots must show retained WAL well under 3640GB and `wal_status = reserved`. Baseline (2026-08-18): all 11 physical member slots ~0; `prd_main_siphon_slot_1` (logical) ~0.6GB. > [!warning] > **Do not proceed** if any slot (particularly the Siphon logical slot) is close to the new cap — investigate the consumer first. ### Change steps - steps to take to execute the change _Estimated Time to Complete_ - approximately 60 minutes, based on the gstg execution on 2026-08-18 (production#22707, ~50 min end-to-end). The scheduled window (time tracking) is 120 minutes: ~60 for execution plus ~60 of post-change monitoring with the EOC before standing down. > [!note] > **Scope:** `gprd-base-db-patroni-main-v17` is a *base role* — the chef change lands on **14 hosts**: the 12 `gprd-patroni-main-v17` cluster members **plus** 2 standalone DR nodes that inherit it ([Danger blast-radius note](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/7829#note_3702610090)): > > | DR node | Patroni scope | > |---|---| > | `postgres-main-dr-archive-v17-01.db.gprd.gitlab.com` | `gprd-patroni-main-archive-v17` | > | `postgres-main-dr-delayed-v17-01-db-gprd.c.gitlab-production.internal` | `gprd-patroni-main-delayed-v17` | > > Both currently report `5200GB` and are intended to receive the new cap. They are single-node Patroni scopes, so the main-cluster reload does **not** reach them — they get their own reloads in step 5. - [x] **1 — Start:** set label ~"change::in-progress" `/label ~change::in-progress` - [x] **2 — Merge** gitlab-com/gl-infra/chef-repo!7829 — `max_slot_wal_keep_size: 5200GB → 3640GB` in `roles/gprd-base-db-patroni-main-v17.json` - [x] **3 — Confirm the Chef Server has the new value** (wait for the chef-repo apply pipeline first; chef-client converges against the *server*, not git): ```shell knife role show gprd-base-db-patroni-main-v17 | grep max_slot_wal_keep_size # expect: 3640GB ``` - [x] **4 — Converge chef on all 14 hosts** (matches the expanded run_list, so the DR nodes are included): ```shell knife ssh 'roles:gprd-base-db-patroni-main-v17' 'sudo chef-client' ``` - [x] **5 — Reload Patroni** — main cluster first, then the two standalone DR scopes: ```shell # on any main cluster member: sudo gitlab-patronictl reload gprd-patroni-main-v17 # on the DR archive node: sudo gitlab-patronictl reload gprd-patroni-main-archive-v17 # on the DR delayed node: sudo gitlab-patronictl reload gprd-patroni-main-delayed-v17 ``` Chef's patroni cookbook may already have reloaded the DR nodes during step 4 — the explicit reloads are confirmation, and are harmless to repeat. - [x] **6 — Verify the new value on all 14 hosts:** ```shell knife ssh 'roles:gprd-base-db-patroni-main-v17' "sudo gitlab-psql -tAc 'show max_slot_wal_keep_size;'" # expect: 3640GB on all 14 lines ``` Then check `sudo gitlab-patronictl list` — no member should show a genuine pending restart. > [!note] > A `Pending restart` flag with reason `dynamic_shared_memory_type: posix->posix` is a known **cosmetic false-positive** (observed during the gstg rehearsal, production#22707). Only `max_slot_wal_keep_size` appearing in the reason column would indicate a real problem. - [x] **7 — Re-run the safety pre-check query** (see pre-execution steps) — all slots still `active` with `wal_status = reserved` - [x] **8 — Done:** set label ~"change::complete" `/label ~change::complete` ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change _Estimated Time to Complete (mins)_ - 15 minutes - [ ] **1 —** Revert gitlab-com/gl-infra/chef-repo!7829 (restores `max_slot_wal_keep_size: 5200GB`) - [ ] **2 —** Re-converge and reload — same commands as change steps 4–5: ```shell knife ssh 'roles:gprd-base-db-patroni-main-v17' 'sudo chef-client' sudo gitlab-patronictl reload gprd-patroni-main-v17 # any main cluster member sudo gitlab-patronictl reload gprd-patroni-main-archive-v17 # on the DR archive node sudo gitlab-patronictl reload gprd-patroni-main-delayed-v17 # on the DR delayed node ``` - [ ] **3 —** Verify `show max_slot_wal_keep_size;` returns `5200GB` on all 14 hosts - [ ] **4 —** Set label ~"change::aborted" `/label ~change::aborted` ## Monitoring ### Key metrics to observe - Metric: Retained WAL per replication slot on the leader (`pg_replication_slots_restart_lsn_bytes`) - Location: [Grafana Explore — retained WAL per slot in GB, gprd patroni-main-v17](https://dashboards.gitlab.net/explore?left=%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22expr%22%3A%22pg_replication_slots_restart_lsn_bytes%7Benv%3D%5C%22gprd%5C%22%2C+fqdn%3D~%5C%22patroni-main-v17-.%2A%5C%22%7D+%2F+1024+%2F+1024+%2F+1024%22%2C%22legendFormat%22%3A%22%7B%7Bslot_name%7D%7D%22%2C%22refId%22%3A%22A%22%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-6h%22%2C%22to%22%3A%22now%22%7D%7D) (`pg_replication_slots_restart_lsn_bytes{env="gprd", fqdn=~"patroni-main-v17-.*"} / 1024 / 1024 / 1024`) - Baseline (2026-08-18): physical member slots ~0; Siphon logical slot ~0.6 GB — all far under the new 3640GB cap - What changes to this metric should prompt a rollback: any slot climbing toward the 3640GB cap after the reload, or a slot getting invalidated (`wal_status` leaving `reserved` in the pre-check SQL) - Metric: `max_slot_wal_keep_size` as seen by PostgreSQL (`pg_settings_max_slot_wal_keep_size_bytes`) - Location: [Grafana Explore — max_slot_wal_keep_size per node in GB](https://dashboards.gitlab.net/explore?left=%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22expr%22%3A%22pg_settings_max_slot_wal_keep_size_bytes%7Benv%3D%5C%22gprd%5C%22%2C+fqdn%3D~%5C%22patroni-main-v17-.%2A%7Cpostgres-main-dr-.%2A%5C%22%7D+%2F+1024+%2F+1024+%2F+1024%22%2C%22legendFormat%22%3A%22%7B%7Bfqdn%7D%7D%22%2C%22refId%22%3A%22A%22%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-6h%22%2C%22to%22%3A%22now%22%7D%7D) (`pg_settings_max_slot_wal_keep_size_bytes{env="gprd", fqdn=~"patroni-main-v17-.*|postgres-main-dr-.*"} / 1024 / 1024 / 1024`) - Verification: all 14 hosts (12 members + 2 DR nodes) step from `5583457484800` (5200GB) to `3908420239360` (3640GB) after chef-client + reload - Metric: Replication lag per node (`pg_replication_lag`) - Location: [Grafana Explore — replication lag, gprd patroni-main-v17](https://dashboards.gitlab.net/explore?left=%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22expr%22%3A%22pg_replication_lag%7Benv%3D%5C%22gprd%5C%22%2C+fqdn%3D~%5C%22patroni-main-v17-.%2A%5C%22%7D%22%2C%22legendFormat%22%3A%22%7B%7Bfqdn%7D%7D%22%2C%22refId%22%3A%22A%22%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-6h%22%2C%22to%22%3A%22now%22%7D%7D) - What changes to this metric should prompt a rollback: sustained lag growth cluster-wide following the reload - Metric: Patroni-main Apdex (GPRD) - Location: https://dashboards.gitlab.net/d/patroni-main/patroni-main-overview?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd - What changes to this metric should prompt a rollback: none expected — a reload of this parameter has no workload impact; investigate before rolling back ## References - [production-engineering#29507 — corrective action + Siphon sign-off](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29507) - [INC-12794 summary](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29507#note_3653636279) - [gstg equivalent — production#22707, completed 2026-08-18](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22707) - Current value: [gprd-base-db-patroni-main-v17.json#L268](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/master/roles/gprd-base-db-patroni-main-v17.json#L268) - Follow-up: carry the new value into the gprd v18 role when it gets created for the PG18 upgrade (tracked in 29507)