# Production Change ## Change Summary The Patroni \[Sec\] cluster currently runs on PostgreSQL 17. During this maintenance, we will upgrade a new standby cluster to PostgreSQL 18, and update the Consul service registration to point GitLab.com applications to the corresponding new Patroni cluster running PG18. Parts of the procedure are similar to the previous PG18 upgrade iteration: * [\[C2\] \[GPRD\] \[2026-06-06 to 2026-06-09\] - Upgrade PostgreSQL to PG18 on Registry](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22144) #### Key Benefits: * Up-to-date PostgreSQL engine will provide increased security and stability. * Performance improvements in PostgreSQL 18 will impact our ability to operate the database at our current scale. #### Upgrade events schedule The CR is executed through the following schedule: > NB: Timings are approximate and worst case scenario. Because there is substantially less data in STG, timings are expected to actually be less than the worst case outlined below. * **Saturday `2026-06-27 01:00Z` - PCL start / Change window opens**: * Stop future deployments and all DDL changes. * Check there are no running migrations * **Saturday `2026-06-27 01:00Z` - Upgrade** * Perform database upgrade in the target cluster (convert to logical replication old PG17 -\> new PG18) * **Saturday `2026-06-27 15:00Z` - Cutover/Switchover** (estimate: start-time + \~6 hours): * **Workload switchover to PG18** (PG18 will become the active database) * Enable reverse replication (new PG17 -\> old PG18) * Start the rollback window * **Tuesday `2026-06-30 15:00Z` - PCL finish / Change window closes**: * End rollback window * End operational lock * Deploys will resume The **PG18 upgrade process will be an online process with `effective zero downtime`** for end-users. The workload should be enqueued during the switchover (a few seconds), but there's a risk of disruption of up to 5 mins if manual intervention is required for the Writer endpoint switchover. This will only affect the Sec database. A Sec-only-PCL is required for the entire change window because logical replication is incompatible with database migrations (DDL). The rollback window allows quick reversion in case of performance regression in the new engine version. ### Postgres Upgrade Rollout Team | Role | Assigned To | |------|-------------| | :wolf: Coordinator | @bshah11 | | :knife: Playbook-Runner | @saadullah707 | | :telephone: Comms-Handler | @bshah11 | | :elephant: DBRE | @saadullah707 @vporalla | | :dolphin: SRE | | | :trophy: Quality | Self-Serve with escalations via PD | | :tophat: IMOC | | | :mega: CMOC | | | :ambulance: EOC | | | :floppy_disk: DBRE Escalation | @alexives @alexander-sosna @dazhu1 | ## Change Details 1. **Services Impacted** - ~"Service::PatroniSec" ~C2 ~Database 2. **Change Technician** - @saadullah707 3. **Change Reviewer** - @vporalla @bshah11 4. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-06-27 01:00 5. **Time tracking** - 3 days 6. **Downtime Component** - Near Zero Customer Downtime (only DDL's will be blocked through the change window) > [!important] > > If your change involves scheduled maintenance, add a step to set and [unset maintenance mode](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/set_maintenance_window.md) per our runbooks. This will make sure SLA calculations adjust for the maintenance period. ## Preparation > [!note] > > The following checklists must be done in advance, before setting the label ~"change::scheduled" ### Change Reviewer checklist ~C4 ~C3 ~C2 ~C1: - [x] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [x] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~"blocks deployments" and/or ~"blocks feature-flags" are applied as necessary. ### Change Technician checklist - [x] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed. - [x] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [x] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access. - [x] This Change Issue is linked to the appropriate Issue and/or Epic - [x] Change has been tested in staging and results noted in a comment on this issue. - [x] A dry-run has been conducted and results noted in a comment on this issue. - [ ] The change execution window respects the [Production Change Lock periods](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#production-change-lock-pcl). - [ ] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` - [ ] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). It is schedule to run every 2 hours. - [x] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - [ ] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed [at least 2 weeks in advance](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#approval). Check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW) to find who will be on-call at the scheduled day and time. ## Detailed steps for the change ### Pre-execution steps > [!note] > > The following steps should be done right at the scheduled time of the change request. The [preparation steps](#preparation) are listed below. - [ ] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [ ] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out. - [ ] The SRE on-call provided approval with the ~eoc_approved label on the issue. - [ ] For ~C1, ~C2, or ~"blocks deployments" change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) that are ~severity::1 or ~severity::2 - [ ] If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change. ### Change steps - steps to take to execute the change _Estimated Time to Complete (mins)_ - 4320 - [x] Set label ~"change::in-progress" `/label ~change::in-progress` > [!note] Detailed implementation plan > > As part of 'worst case' planning, we assume that GitLab.com would be unavailable during the execution of database upgrade change requests and use ops.gitlab.net for the detailed instructions. > * [x] ops.gitlab.net implementation plan: https://ops.gitlab.net/gitlab-com/gl-infra/db-migration/-/work_items/107 > * [ ] ops.gitlab.net implementation backup plan: https://ops.gitlab.net/gitlab-com/gl-infra/db-migration/-/work_items/108 - [x] Perform database upgrade in the target cluster (convert logical replication old PG17 -\> new PG18) per the ops implementation plan above - [x] Perform Cutover/Switchover — workload switchover to PG18 per the ops implementation plan - [x] Enable reverse replication (PG17 -\> PG18) and start rollback window - [x] Monitor for performance regressions until end of rollback window - [ ] End rollback window and confirm change is stable - [x] Set label ~"change::complete" `/label ~change::complete` ## Rollback ### Rollback scenarios in case of incident There are 3 approaches depending on when an incident occurs: * **Before Switchover**: Abort the CR; the database remains on PG17. * **After Switchover**: * Anticipate the `end rollback window` but keep the database on PG18. * Or rollback the upgrade and return the database to PG17. In any case, Deployments will resume. For non-production environments like GSTG, no full on-call coverage is planned — at least 1 change technician will be available during the change window. > Note: The CR only needs to be aborted if the incident fix requires a DDL (CREATE, ALTER, DROP, or REINDEX). A DBRE must evaluate the fix before deciding. ### Rollback steps - steps to be taken in the event of a need to rollback this change _Estimated Time to Complete (mins)_ - 120 - [ ] Rollback steps are accurately stated on the OPS runbook - [ ] Set label ~"change::aborted" `/label ~change::aborted` ## Monitoring ### Key metrics to observe - Metric: Monitor the Patroni Sec Dashboard - Location: https://dashboards.gitlab.net/d/patroni-sec-main/patroni-sec-overview?orgId=1&from=now-6h%2Fm&to=now%2Fm&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd - Look for all the KPI on patroni service - Metrics: Monitor the PG18 Upgrade Dashboard * Location: https://dashboards.gitlab.net/d/postgresql-pg-upgrade/postgresql3a-postgres-upgrade-using-logical?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-cluster=sec&var-source_version=17&var-destination_version=18&from=now-1h&to=now&refresh=5m&timezone=utc * Monitor the logical Replication Lag.