# Production Change ## Change Summary The Patroni \[CI\] cluster currently runs on PostgreSQL 17. During this maintenance, we will upgrade a new standby cluster to PostgreSQL 18, and update the Consul service registration to point GitLab.com applications to the corresponding new Patroni cluster running PG18. This is a GPRD (Production) CI upgrade, mirroring the approach used in: * [\[C2\] \[GSTG\] \[2026-07-04\] - Upgrade PostgreSQL to PG18 on CI](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22398) * ops.gitlab.net implementation plan: https://ops.gitlab.net/gitlab-com/gl-infra/db-migration/-/work_items/109 * ops.gitlab.net implementation backup plan: https://ops.gitlab.net/gitlab-com/gl-infra/db-migration/-/work_items/110 #### Key Benefits: * Up-to-date PostgreSQL engine will provide increased security and stability. * Performance improvements in PostgreSQL 18 will impact our ability to operate the database at our current scale. #### Upgrade events schedule The CR is executed through the following schedule: * **Friday `2026-07-10 22:00Z` - PCL start / Change window opens**: * Stop future deployments and all DDL changes. * Check there are no running migrations * **Saturday `2026-07-11 01:00Z` - Upgrade** * Perform database upgrade in the target cluster (convert to logical replication old PG17 -\> new PG18) * **Saturday`2026-07-11 13:00Z` - Cutover/Switchover** (estimate: start-time + \~6 hours): * **Workload switchover to PG18** (PG18 will become the active database) * Enable reverse replication (new PG17 -\> old PG18) * Start the rollback window * **Tuesday `2026-07-14 14:00Z` - PCL finish / Change window closes**: * End rollback window * End operational lock * Deploys will resume The **PG18 upgrade process will be an online process with `effective zero downtime`** for end-users. The workload should be enqueued during the switchover (a few seconds), but there's a risk of disruption of up to 5 mins if manual intervention is required for the Writer endpoint switchover. This will only affect the CI database. A CI-only-PCL is required for the entire change window because logical replication is incompatible with database migrations (DDL). ### Postgres Upgrade Rollout Team | Role | Assigned To | |------|-------------| | :wolf: Coordinator | @vporalla @saadullah707 | | :knife: Playbook-Runner | @saadullah707 | | :telephone: Comms-Handler | @bshah11 | | :elephant: DBRE | @bshah11 @vporalla @saadullah707 | | :dolphin: SRE | TBD | | :trophy: Quality | Self-Serve with escalations via PD | | :tophat: IMOC | TBD | | :mega: CMOC | TBD | | :ambulance: EOC | TBD | | :floppy_disk: DBRE Escalation | @alexives @dazhu1 | ## Change Details 1. **Services Impacted** - ~"Service::PatroniCI" ~C2 ~Database 2. **Change Technician** - @saadullah707 @vporalla 3. **Change Reviewer** - @bshah 4. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-07-10 22:00 5. **Time tracking** - 75 hours 6. **Downtime Component** - Near Zero Customer Downtime (only DDL's will be blocked through the change window) > [!important] > > If your change involves scheduled maintenance, add a step to set and [unset maintenance mode](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/set_maintenance_window.md) per our runbooks. This will make sure SLA calculations adjust for the maintenance period. ## Preparation > [!note] > > The following checklists must be done in advance, before setting the label ~"change::scheduled" ### Change Reviewer checklist ~C4 ~C3 ~C2 ~C1: - [ ] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [ ] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~"blocks deployments" and/or ~"blocks feature-flags" are applied as necessary. ### Change Technician checklist - [x] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed. - [x] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [x] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access. - [x] This Change Issue is linked to the appropriate Issue and/or Epic - [x] Change has been tested in staging and results noted in a comment on this issue. - [x] A dry-run has been conducted and results noted in a comment on this issue. - [ ] The change execution window respects the [Production Change Lock periods](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#production-change-lock-pcl). - [x] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` - [ ] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). It is schedule to run every 2 hours. - [x] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - [ ] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed [at least 2 weeks in advance](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#approval). Check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW) to find who will be on-call at the scheduled day and time. ## Detailed steps for the change ### Pre-execution steps > [!note] > > The following steps should be done right at the scheduled time of the change request. The [preparation steps](#preparation) are listed below. - [ ] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [ ] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out. - [ ] The SRE on-call provided approval with the ~eoc_approved label on the issue. - [ ] For ~C1, ~C2, or ~"blocks deployments" change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) that are ~severity::1 or ~severity::2 - [ ] If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change. ### Change steps - steps to take to execute the change _Estimated Time to Complete (mins)_ - 4380 - [x] Set label ~"change::in-progress" `/label ~change::in-progress` > [!note] Detailed implementation plan > > As part of 'worst case' planning, we assume that GitLab.com would be unavailable during the execution of database upgrade change requests and use ops.gitlab.net for the detailed instructions. > * [x] ops.gitlab.net implementation plan: https://ops.gitlab.net/gitlab-com/gl-infra/db-migration/-/work_items/111 > * [ ] ops.gitlab.net implementation backup plan: [https://ops.gitlab.net/gitlab-com/gl-infra/db-migration/-/work_items/11](https://ops.gitlab.net/gitlab-com/gl-infra/db-migration/-/work_items/110)2 - [ ] Perform database upgrade in the target cluster (convert logical replication old PG17 -\> new PG18) per the ops implementation plan above - [ ] Perform Cutover/Switchover — workload switchover to PG18 per the ops implementation plan - [ ] Enable reverse replication (PG17 -\> PG18) and start rollback window - [ ] Monitor for performance regressions until end of rollback window - [ ] End rollback window and confirm change is stable - [ ] Set label ~"change::complete" `/label ~change::complete` ## Rollback ### Rollback scenarios in case of incident There are 3 approaches depending on when an incident occurs: * **Before Switchover**: Abort the CR; the database remains on PG17. * **After Switchover**: * Anticipate the `end rollback window` but keep the database on PG18. * Or rollback the upgrade and return the database to PG17. In any case, Deployments will resume. > Note: The CR only needs to be aborted if the incident fix requires a DDL (CREATE, ALTER, DROP, or REINDEX). A DBRE must evaluate the fix before deciding. ### Rollback steps - steps to be taken in the event of a need to rollback this change _Estimated Time to Complete (mins)_ - 120 - [ ] Rollback steps are accurately stated on the OPS runbook - [ ] Set label ~"change::aborted" `/label ~change::aborted` ## Monitoring ### Key metrics to observe - Metric: Monitor the Patroni CI Dashboard - Location: https://dashboards.gitlab.net/d/patroni-ci-main/patroni-ci-overview?orgId=1&from=now-6h%2Fm&to=now%2Fm&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd - Look for all the KPI on patroni service - Metrics: Monitor the PG18 Upgrade Dashboard * Location: https://dashboards.gitlab.net/d/postgresql-pg-upgrade/postgresql3a-postgres-upgrade-using-logical?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-cluster=ci&var-source_version=17&var-destination_version=18&from=now-1h&to=now&refresh=5m&timezone=utc * Monitor the logical Replication Lag.