Skip to content

GitLab Geo 15.11 Database Upgrade failing

Summary

After upgrading PostgreSQL when using Geo (GitLab version 15.11.3), from Postgres v12.12 to 13.8 the secondary Geo database initializes with version 12 instead version 13. This results in a system identifier mismatch between the primary and secondary databases, leading to a failure in database replication.

Steps to reproduce

  1. Attempt to upgrade Postgres to version 13 on a GitLab Geo secondary node running version 15.11.
  2. Observe the postgress logs for database identifier errors.

What is the current bug behavior?

The upgrade process for Postgres does not successfully upgrade to version 13, and the data directory on the Geo secondary is initialized with version 12. The system identifier of the secondary does not match the primary, causing fatal errors in database replication.

What is the expected correct behavior?

During the upgrade process, the Geo secondary should initialize the data directory with Postgres version 13, ensuring the system identifiers of the primary and secondary match and database replication occurs without error.

Relevant logs

Relevant logs 
2023-05-12_04:34:53.49477 DETAIL:  The primary's identifier is 7232145530001217774, the standby's identifier is 7164224136944485225.
2023-05-12_04:34:58.55881 FATAL:  database system identifier differs between the primary and standby

Details of package version

Provide the package version installation details

Environment details

  • Upgrade from 15.10.2 to 15.11.2.
  • Single node Geo installation

Possible Workaround:

  1. Initialize PostgreSQL with version 13. For that you need to set postgresql['version'] in the gitlab.rb file:
postgresql['version'] = 13
  1. Move/Rename the data directory. By default, it's location is: /var/opt/gitlab/geo-postgresql/data
  2. Reconfigure GitLab the Geo Secondary node to do the required configuration changes and initialize with v13
sudo gitlab-ctl reconfigure
  1. Restart the database replication on the Geo secondary database with the following command:
sudo gitlab-ctl replicate-geo-database --slot-name=<SECONDARY_SLOT_NAME> --host=<PRIMARY_HOST_NAME>
Edited by David Wainaina