# Production Change ## Change Summary Upgrade the packaged PostgreSQL on `dev.gitlab.org` from major version **16 to 17**. Omnibus made PostgreSQL 17 a hard requirement, so nightly `gitlab-ce` upgrades on `dev.gitlab.org` now fail the preinstall check until Postgres is upgraded to 17. This change unblocks the daily upgrades of `dev.gitlab.org`. - Tracking issue: gitlab-com/gl-infra/delivery#22075 - Runbook: https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/postgres-upgrades-dev.md - WAL-G prefix MR (gated on this upgrade): gitlab-com/gl-infra/chef-repo!7449 ## Backup data locations The following backups protect this change, in the order they would be used for recovery: | Backup | Location | |--------|----------| | Pre-upgrade machine image (created during this change) | GCE machine image `cr22234` in project `gitlab-dev-1` ([console](https://console.cloud.google.com/compute/machineImages?project=gitlab-dev-1)), sourced from instance `dev-1-01-sv-dev-1` (zone `us-east1-c`) | | Retained old PostgreSQL data directory (kept by `pg-upgrade`) | `/var/opt/gitlab/postgresql/data.16` on `dev.gitlab.org` (version marker: `/var/opt/gitlab/postgresql-version.old`) | | WAL-G PITR backups (PG16, pre-upgrade lineage) | `gs://gitlab-dev-1-postgres-backup/pitr-walg-dev-pg16/` (project `gitlab-dev-1`) | | WAL-G PITR backups (PG17, post-upgrade lineage, after gitlab-com/gl-infra/chef-repo!7449 merges) | `gs://gitlab-dev-1-postgres-backup/pitr-walg-dev-pg17/` | The existing `pg16` WAL-G archives are kept untouched after the upgrade, so they remain available for point-in-time recovery during the post-upgrade window. ## Change Details 1. **Services Impacted** - ~&quot;Service::Postgres&quot; on dev.gitlab.org 2. **Change Technician** - `@dat.tang.gitlab` 3. **Change Reviewer** - `@mbhadrannavar` 4. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-06-03 09:00 5. **Time tracking** - 180 minutes 6. **Downtime Component** - No downtime for gitlab.com SaaS. dev.gitlab.org is placed in maintenance mode during the upgrade. > [!important] > > If your change involves scheduled maintenance, add a step to set and [unset maintenance mode](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/monitoring/set_maintenance_window.md) per our runbooks. This will make sure SLA calculations adjust for the maintenance period. ## Preparation > [!note] > > The following checklists must be done in advance, before setting the label ~&quot;change::scheduled&quot; ### Change Reviewer checklist ~C4 ~C3 ~C2 ~C1: - [x] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [x] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~&quot;blocks deployments&quot; and/or ~&quot;blocks feature-flags&quot; are applied as necessary. ### Change Technician checklist - [x] The [Change Criticality](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#change-criticalities) has been set appropriately and requirements have been reviewed. - [x] The [change plan](#detailed-steps-for-the-change) is technically accurate. - [x] The [rollback plan](#rollback) is technically accurate and detailed enough to be executed by anyone with access. - [x] This Change Issue is linked to the appropriate Issue and/or Epic - [x] Change has been tested in staging and results noted in a comment on this issue. * [x] https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22075#note_3411358763 - [ ] A dry-run has been conducted and results noted in a comment on this issue. - [x] The change execution window respects the [Production Change Lock periods](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#production-change-lock-pcl). - [x] Once all boxes above are checked, mark the change request as scheduled: `/label ~"change::scheduled"` - [ ] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). It is schedule to run every 2 hours. - [ ] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - [x] For ~C1, ~C2, or ~&quot;blocks deployments&quot; change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [ ] For ~C1 change issues or ~C2 change issues happening during weekend, SREs on-call must be informed [at least 2 weeks in advance](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#approval). Check [the incident.io GitLab.com Production EOC schedule](https://app.incident.io/gitlab/on-call/schedules/01K5YWAGZ7YCQGAG7ATQ9XQWHW) to find who will be on-call at the scheduled day and time. ### Preparation steps - steps to take in advance of the change These steps need to be done successfully _in advance_ to make sure that the change can be performed. It's recommended to complete them a few days in advance from the change request due date. - [x] Coordinate with the `@release-managers` to plan for the auto-deployments to be paused during the change window. - [x] Test local gcloud access: `gcloud config get project` - [x] Test SSH access to the dev instance: `ssh dev.gitlab.org` - [x] Perform the [preparation steps for upgrading packaged PostgreSQL server](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server). No need to manually upgrade and run `sudo gitlab-ctl reconfigure` during these steps as it should be covered by deploying the nightly builds. - [x] Confirm there is sufficient disk space for two copies of the database. Check the database size and available space: ```bash sudo du -sh /var/opt/gitlab/postgresql/data sudo df -h ``` If the partition where the database resides doesn't have enough space, pass `--tmp-dir $DIR` to the `pg-upgrade` command. If using a custom temporary directory, ensure it has the correct ownership: `sudo chown gitlab-psql:gitlab-psql $DIR`. - [x] Result: the DB size is 287GB, while the available storage is 366GB ``` dattang@dev-1-01-sv-dev-1.c.gitlab-dev-1.internal:~$ sudo du -sh /var/opt/gitlab/postgresql/data 287G /var/opt/gitlab/postgresql/data dattang@dev-1-01-sv-dev-1.c.gitlab-dev-1.internal:~$ sudo df -h Filesystem Size Used Avail Use% Mounted on /dev/root 969G 604G 366G 63% / tmpfs 63G 9.1M 63G 1% /dev/shm tmpfs 26G 200M 25G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sda15 105M 6.1M 99M 6% /boot/efi /dev/sdb 49G 11G 39G 22% /var/log tmpfs 13G 4.0K 13G 1% /run/user/1132 ``` - [x] Prepare a MR to move dev.gitlab.org wal-g backup location to the new version. This is already prepared: gitlab-com/gl-infra/chef-repo!7449. Get the MR approved, and keep it in `draft` mode until the actual change day. ## Detailed steps for the change ### Pre-execution steps - [x] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done - [ ] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out. - [ ] The SRE on-call provided approval with the ~eoc_approved label on the issue. - [x] For ~C1, ~C2, or ~&quot;blocks deployments&quot; change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [x] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) that are ~severity::1 or ~severity::2 - [ ] If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change. ### Change steps - steps to take to execute the change _Estimated Time to Complete (mins)_ - 180 minutes - [x] Set label ~&quot;change::in-progress&quot; `/label ~change::in-progress` - [x] Notify EOC and current RM's of the start of this maintenance in slack, leverage handles `@sreoncall` and `@release-managers` - [x] Pause Auto-Deploy (in `#f_upcoming_release`): `/chatops gitlab run auto_deploy pause` - [x] Configure your local `gcloud` CLI to leverage the `gitlab-dev` project: ```bash gcloud config set project gitlab-dev-1 # Check again after setting gcloud config get project ``` - [ ] Open an ssh session to the `dev` instance: `ssh dev.gitlab.org` - [x] Stop chef: `sudo systemctl stop chef-client` - [x] Comment out the `wal-g backup` cron: `sudo -u gitlab-psql crontab -e` (it's the only item in there) - [x] Enable maintenance mode. Open a rails console on the dev instance: `sudo gitlab-rails c` ```ruby ::Gitlab::CurrentSettings.update!(maintenance_mode: true) ::Gitlab::CurrentSettings.update!(maintenance_mode_message: "Postgres Upgrade: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/22234") ``` - [x] From a terminal on your local machine: Take Snapshot of Data Disk for the dev instance: `gcloud compute machine-images create cr22234 --source-instance=dev-1-01-sv-dev-1 --source-instance-zone=us-east1-c`. The image name `cr22234` follows the convention `cr<issue_number>`, where `22234` is the IID of this change request (gitlab-com/gl-infra/production#22234), tying the backup image to the CR that created it. If this command doesn't work as intended, you can always create the machine image directly using the gcloud console. * [x] After it is done, you should see it on the GCP Console https://console.cloud.google.com/compute/machineImages?referrer=search&project=gitlab-dev-1 - [x] Delete the auto-upgrade skip file: `sudo rm /etc/gitlab/disable-postgresql-upgrade` - [x] Upgrade the database (timeout of 3 hours): (it is normal to take around 1 hour) ```bash sudo gitlab-ctl pg-upgrade -V 17 --timeout=3h ``` If there was an error in the output while running the `ANALYZE` step, follow the [official documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to determine whether `ANALYZE` should be run manually. - [x] Verify that the psql binary and the data versions are updated to the new version: ```bash $ /opt/gitlab/embedded/bin/psql --version psql (PostgreSQL) 17.8 $ sudo cat /var/opt/gitlab/postgresql/data/PG_VERSION 17 ``` - [x] Verify that the dev instance is online, and shows under the [admin page](https://dev.gitlab.org/admin) that the postgres version is updated. {width=404 height=82} - [x] Determine whether `ANALYZE` should be run manually to regenerate database statistics: ```bash sudo gitlab-psql -c "SELECT relname, last_analyze, last_autoanalyze FROM pg_stat_user_tables WHERE last_analyze IS NULL AND last_autoanalyze IS NULL;" ``` If the query above returned any rows, run `ANALYZE` manually: ```bash sudo gitlab-psql -c 'SET statement_timeout = 0; ANALYZE VERBOSE;' ``` - [x] Following [Monitoring](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22234#monitoring) to verify if the instance is running well - [x] After you have verified that your GitLab instance is running correctly, you can clean up the old database files: ```bash sudo rm -rf /var/opt/gitlab/postgresql/data.16 sudo rm -f /var/opt/gitlab/postgresql-version.old ``` - [x] Add the file back to disable pg upgrades: `sudo touch /etc/gitlab/disable-postgresql-upgrade` - [x] Remove maintenance mode. Open a rails console on the dev instance: `sudo gitlab-rails c` ```ruby ::Gitlab::CurrentSettings.update!(maintenance_mode: false) ``` - [x] Merge the MR to move dev.gitlab.org wal-g backup location to the new version (gitlab-com/gl-infra/chef-repo!7449). After merging, ensure the merge commit was mirrored to `ops` (there will be a bot comment with a pipeline). - [x] Start and Run Chef: `sudo systemctl start chef-client; sudo chef-client` - [x] Validate our cron change from earlier has been reverted: `sudo -u gitlab-psql crontab -l` - [x] Unpause Auto-Deploy (in `#f_upcoming_release`): `/chatops gitlab run auto_deploy unpause` - [x] Notify EOC and current RM's of the completion of this maintenance in slack, leverage handles `@sreoncall` and `@release-managers` - [x] Set label ~&quot;change::complete&quot; `/label ~change::complete` ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change _Estimated Time to Complete (mins)_ - 20 minutes In case we need to rollback, follow all three sections in order: rollback, verify, and cleanup. #### Rollback - [ ] If the `pg-upgrade` script encounters a fatal error, it will attempt to auto-rollback. - If the auto-rollback worked, perform the verify steps below. - [ ] If the auto-rollback did not work, we need to rollback manually ([official docs for revert](https://docs.gitlab.com/omnibus/settings/database.html#revert-packaged-postgresql-server-to-the-previous-version)). The old version is `16` (also in `/var/opt/gitlab/postgresql-version.old`): ```bash gitlab-ctl revert-pg-upgrade -V 16 ``` - [ ] If both above options fail, we need to restore from the data disk. - Follow the [official docs for detaching and reattaching boot disks](https://cloud.google.com/compute/docs/disks/detach-reattach-boot-disk). - Make appropriate changes to [`config-mgmt`](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/) after this is completed. #### Verify the rollback - [ ] Verify that the psql binary and the data versions still show the old version: ```bash /opt/gitlab/embedded/bin/psql --version cat /var/opt/gitlab/postgresql/data/PG_VERSION ``` - [ ] Verify that the dev instance is back online, and shows under the [admin page](https://dev.gitlab.org/admin) that the postgres version is back to the old version. - [ ] Gather diagnostic information as needed, and add the information to the change request issue comments for future reference. #### Cleanup after rollback - [ ] Add the file back to disable pg upgrades: `sudo touch /etc/gitlab/disable-postgresql-upgrade` - [ ] Remove maintenance mode. Open a rails console on the dev instance: `sudo gitlab-rails c` ```ruby ::Gitlab::CurrentSettings.update!(maintenance_mode: false) ``` - [ ] Start and Run Chef: `sudo systemctl start chef-client; sudo chef-client` - [ ] Validate our cron change from earlier has been reverted: `sudo -u gitlab-psql crontab -l` - [ ] Unpause Auto-Deploy (in `#f_upcoming_release`): `/chatops gitlab run auto_deploy unpause` - [ ] Notify EOC and current RM's of the completion of this maintenance in slack, leverage handles `@sreoncall` and `@release-managers` - [ ] Set label ~&quot;change::aborted&quot; `/label ~change::aborted` ## Monitoring ### Key metrics to observe - Metric: dev.gitlab.org availability - Location: https://dev.gitlab.org/ and the [admin page](https://dev.gitlab.org/admin) - What changes to this metric should prompt a rollback: dev.gitlab.org is not accessible after the upgrade. - Metric: `HTTP500` errors related to database connectivity - Location: [log query](https://nonprod-log.gitlab.net/app/r/s/8OZMs) - What changes to this metric should prompt a rollback: a sustained increase in DB-connectivity `HTTP500` errors (note that occasional `HTTP500` errors are normal).