## Summary Upgrade of all GitLab.com shared SaaS PostgreSQL databases from PG17 to PG18. **DRIs:** @vporalla, @saadullah707 and @bshah11 > **Scope:** This epic covers the **PostgreSQL engine upgrade only** (PG17 → PG18). An OS upgrade (Ubuntu 22.04 → 24.04) is **not included** in this cycle due to dependencies on the Chef/Cinc migration, tracked separately in [Epic #1885 — Migrate off Chef 14](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1885). The OS upgrade will be considered for a future cycle once those dependencies are resolved. > This epic replaces [Epic #54 — Postgres 18 upgrade](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/dbo/-/work_items/54), which will be closed in favor of this one. --- ## Expected Impact **PG18 key improvements:** - **LWLock contention relief** — configurable fast-path locking (`max_locks_per_transaction`) is expected to materially alleviate the `LockMgr` LWLock problem we have been experiencing. Early evaluation in [Epic #52 — Evaluate the impact of PG18](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/dbo/-/work_items/52) is promising. - **Async I/O (AIO)** — improved performance for large scans, index scans, and VACUUM. AIO settings evaluation tracked in [#714](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/714). - **General performance improvements** — see [PG18 release notes](https://www.postgresql.org/docs/18/release-18.html) for the full list. --- ## Upgrade Structure The work is organized into four sub-epics: 1. **[PG18] Upgrade Process Improvements, Automation & Planning** — streamlining the upgrade process (reducing 80+ manual steps), automation tooling, playbook updates for PG18, planning and scheduling 2. **[PG18] Staging Upgrade (GSTG)** — upgrade all GSTG databases to PG18 3. **[PG18] Production Upgrade (GPRD)** — rolling upgrade of all GPRD databases (Sec → CI/Registry → Main) 4. **[PG18] Post-Upgrade Tasks** — tuning, cleanup, retrospective --- ## Constraints & Risks - **OS upgrade excluded** — Ubuntu 22.04 → 24.04 is not part of this cycle. See [Epic #1885](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1885). - **Early version risk** — we will follow our standard approach of waiting for a stable minor release (18.1+) before upgrading production. - **Performance regression** — query plan changes are possible with any major version upgrade. Mitigation: thorough testing in GSTG and db-benchmarking before GPRD. - **Extension compatibility** — all extensions must be validated against PG18 before proceeding. --- ## Exit Criteria All GitLab.com SaaS databases (Sec, CI, Registry, Main) upgraded to PG18 in production. --- ## Current Status **As of 2026-08-19**, all staging clusters and 2 of 4 production clusters have been successfully upgraded to PG18. The GPRD CI upgrade was attempted on 2026-07-11 but aborted due to performance issues observed on PG18. A mock upgrade is scheduled for 2026-08-22 to capture data, perform RCA, and validate application-level fixes before reattempting the production upgrade. ### Cluster Upgrade Status | Environment | Cluster | Status | Upgrade Date | CR | |-------------|---------|--------|--------------|-----| | **GSTG** | **Registry** | ✅ Upgraded | 2026-05-14 | [#22032](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22032) | | **GSTG** | **Sec** | ✅ Upgraded | 2026-06-24 | [#22298](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22298) | | **GSTG** | **CI** | ✅ Upgraded | 2026-07-06 | [#22398](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22398) | | **GSTG** | **Main** | ✅ Upgraded | 2026-08-08 | [#22539](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22539) | | **GPRD** | **Registry** | ✅ Upgraded | 2026-06-06 | [#22144](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22144) | | **GPRD** | **Sec** | ✅ Upgraded | 2026-06-27 | [#22363](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22363) | | **GPRD** | **CI** | ⚠️ Aborted — Pending Mock Upgrade | 2026-07-11 (aborted) | [#22444](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22444) | | **GPRD** | **Main** | 🔜 Pending | TBD | — | ### GPRD CI — Current Situation & Path Forward The **GPRD CI** upgrade was initiated on **2026-07-11** ([CR #22444](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22444)) but was **aborted due to performance degradation observed on PG18** — specifically query plan regressions identified in [#755 — Plan flip on queries for PG17 vs PG18 (GPRD CI cluster)](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/755). A second attempt was also aborted ([CR #22507](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22507)). Meeting notes, team discussions, decisions, and next steps are captured in the [GPRD CI PG18 Upgrade — Working Doc](https://docs.google.com/document/d/1REV-Gl5B9PC1qhb3bX8kxYz-e7eXaVH1KUW2A_OB6YI/edit?usp=sharing). The path forward: 1. **2026-08-22 — Mock upgrade on GPRD CI** to capture performance data and identify the root cause of the regression (see [#755](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/755)) 2. **RCA & application code fix** — implement the fix on PG17 to address the identified query plan changes 3. **Validation mock upgrade** — re-run the mock upgrade to confirm the fix resolves the performance degradation on PG18 with no regression 4. **GPRD CI upgrade** — proceed with the production upgrade once validated 5. **GPRD Main upgrade** — planned after GPRD CI is successfully upgraded --- ## References - [Epic #52 — Evaluate the impact of PG18](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/dbo/-/work_items/52) - [#681 — Streamlining the current PostgreSQL upgrade process](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/681) - [#714 — Evaluate AIO settings on Hyperdisks](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/work_items/714) - [Epic #54 — Postgres 18 upgrade (replaced)](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/dbo/-/work_items/54) - [Epic #14 — PG17 + Ubuntu 22.04 Upgrade (reference)](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/dbo/-/work_items/14) - [Epic #1239 — PG16 Upgrade (reference)](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1239) - [Epic #1885 — Migrate off Chef 14 (OS upgrade dependency)](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1885) - [PG18 release notes](https://www.postgresql.org/docs/18/release-18.html)