Skip to content

HA with Geo upgrade from 12.0.9 to 12.1.9 does not upgrade PostgreSQL

Summary

The Geo team recently started conducting upgrade calls to a) understand the correct steps to upgrade an HA installation including Geo b) test upgrades between minor GitLab versions.

During the upgrade from 12.0.9 to 12.1.9 we expected the PostgreSQL database to be updated from 9.6.11 to 10.7; however, no PostgreSQL upgrade was performed. The resulting, healthy HA + Geo Configuration is still running PostgreSQL 9.6.11

It is unclear what the instructions are to actually upgrade PostgreSQL and/or if this is expected.

Steps to reproduce

We followed steps as described in !3577 (merged) These steps are new and we never had any official documentation for how to upgrade HA + Geo.

What is the current bug behavior?

We follow steps outlined in !3577 (merged) but PostgreSQL remains at version 9.6.11.

What is the expected correct behavior?

PostgresSQL should be updated to 10.7 unless sudo touch /etc/gitlab/disable-postgresql-upgrade is created. See https://docs.gitlab.com/ee/administration/geo/replication/version_specific_updates.html#updating-to-gitlab-121

Relevant logs

Details of package version

Environment details

We are using an HA cluster in the following configuration:

Primary (US)

VM Instance
ha-postgresql
ha-redis
ha-gitaly
ha-app-01
ha-app-02
ha-load-balancer

Secondary (EU)

VM Instance
geo-postgresql
geo-tracking
geo-redis
geo-gitaly
geo-app-01
geo-app-02
geo-load-balancer

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb` 

PRIMARY HA APP01



external_url 'http://primary-ha.gogitlab.ml'
roles ['application_role']
nginx['enable'] = true
gitaly['enable'] = false
git_data_dirs({
  'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.142.0.11
:8075' }
})
gitlab_rails['gitaly_token'] = ''
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '10.142.0.9' # IP/hostname of database server
gitlab_rails['db_password'] = ''
gitlab_rails['redis_port'] = '6379'
gitlab_rails['redis_host'] = '10.142.0.10' # IP/hostname of Redis server
gitlab_rails['redis_password'] = ''

roles ['geo_primary_role']






PRIMARY HA APP02




external_url 'http://primary-ha.gogitlab.ml'
roles ['application_role']
nginx['enable'] = true
gitaly['enable'] = false
git_data_dirs({
  'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.142.0.11
:8075' }
})
gitlab_rails['gitaly_token'] = ''
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '10.142.0.9' # IP/hostname of database server
gitlab_rails['db_password'] = ''
gitlab_rails['redis_port'] = '6379'
gitlab_rails['redis_host'] = '10.142.0.10' # IP/hostname of Redis server
gitlab_rails['redis_password'] = ''
roles ['geo_primary_role']
gitlab_rails['auto_migrate'] = false





PRIMARY HA GITALY




external_url 'http://gitlab.example.com'
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
gitlab_rails['rake_cache_clear'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['internal_api_url'] = 'http://primary-ha.gogitlab.ml'
gitaly['listen_addr'] = "10.142.0.11:8075"
gitaly['auth_token'] = ''
gitaly['storage'] = [
  { 'name' => 'default', 'path' => '/var/opt/gitlab/git-data' },
]





PRIMARY HA PoSTGRESQL




external_url 'http://primary-ha-postgresql.gogitlab.ml'
roles ['geo_primary_role', 'postgres_role']
repmgr['enable'] = false
consul['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
pgbouncer_exporter['enable'] = false
redis_exporter['enable'] = false
gitlab_monitor['enable'] = false
postgresql['listen_address'] = '10.142.0.9'
postgresql['port'] = 5432
postgresql['sql_user_password'] = ''
postgresql['trust_auth_cidr_addresses'] = %w(10.142.0.12/32 10.142.0.13/32)
postgresql['md5_auth_cidr_addresses'] = ['10.142.0.9/32', '10.164.0.30/32']
gitlab_rails['auto_migrate'] = false
postgresql['max_replication_slots'] = 1





PRIMARY HA REDIS




external_url 'http://primary-ha-redis.gogitlab.ml'
redis['enable'] = true
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
unicorn['enable'] = false
postgresql['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
pgbouncer_exporter['enable'] = false
gitlab_monitor['enable'] = false
gitaly['enable'] = false
redis['bind'] = '10.142.0.10'
redis['port'] = 6379
redis['password'] = 'password'
gitlab_rails['auto_migrate'] = false





SECONDARY HA APP01




external_url 'http://secondary-ha.gogitlab.ml'
roles ['geo_secondary_role', 'application_role']
nginx['enable'] = true
gitlab_rails['auto_migrate'] = false
geo_secondary['db_host'] = '10.164.0.31'
geo_secondary['db_password'] = ''
geo_postgresql['enable'] = false
gitlab_rails['db_host'] = '10.164.0.30'
gitlab_rails['db_password'] = ''
gitlab_rails['redis_host'] = '10.164.0.32'
gitlab_rails['redis_password'] = ''
gitaly['enable'] = false
git_data_dirs({
  'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.164.0.33
:8075' }
})
gitlab_rails['gitaly_token'] = ''





SECONDARY HA APP02




external_url 'http://secondary-ha.gogitlab.ml'
roles ['geo_secondary_role', 'application_role']
gitlab_rails['auto_migrate'] = false
nginx['enable'] = true
geo_secondary['db_host'] = '10.164.0.31'
geo_secondary['db_password'] = ''
geo_postgresql['enable'] = false
gitlab_rails['db_host'] = '10.164.0.30'
gitlab_rails['db_password'] = ''
gitlab_rails['redis_host'] = '10.164.0.32'
gitlab_rails['redis_password'] = ''
gitaly['enable'] = false
git_data_dirs({
  'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.164.0.33
:8075' }
})
gitlab_rails['gitaly_token'] = ''
geo_secondary['auto_migrate'] = false





SECONDARY HA GITALY




external_url 'http://gitlab.example.com'
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
gitlab_rails['rake_cache_clear'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['internal_api_url'] = 'http://secondary-ha.gogitlab.ml'
gitaly['listen_addr'] = "10.164.0.33:8075"
gitaly['auth_token'] = ''
gitaly['storage'] = [
  { 'name' => 'default', 'path' => '/var/opt/gitlab/git-data' },
]
geo_secondary['auto_migrate'] = false





SECONDARY HA POSTGRESQL




external_url 'http://secondary-ha-postgresql.gogitlab.ml'
roles ['postgres_role']
repmgr['enable'] = false
postgresql['listen_address'] = '10.164.0.30'
postgresql['md5_auth_cidr_addresses'] = ['10.164.0.30/32', '10.164.0.31/32', '10.164.0.34/32'
, '10.164.0.35/32']
postgresql['hot_standby'] = 'on'
postgresql['sql_user_password'] = ''
gitlab_rails['db_password'] = ''
geo_postgresql['enable'] = false
gitlab_rails['auto_migrate'] = false
geo_secondary['auto_migrate'] = false





SECONDARY HA REDIS




external_url 'http://secondary-ha-redis.gogitlab.ml'
redis['enable'] = true
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
unicorn['enable'] = false
postgresql['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
pgbouncer_exporter['enable'] = false
gitlab_monitor['enable'] = false
gitaly['enable'] = false
redis['bind'] = '10.164.0.32'
redis['port'] = 6379
redis['password'] = ''
gitlab_rails['auto_migrate'] = false
geo_secondary['auto_migrate'] = false





SECONDARY HA TRACKING




external_url 'http://secondary-ha-tracking.gogitlab.ml'
geo_postgresql['enable'] = true
geo_postgresql['listen_address'] = '10.164.0.31'
geo_postgresql['sql_user_password'] = ''
geo_secondary['db_fdw'] = true
geo_postgresql['fdw_external_password'] = ''
geo_postgresql['md5_auth_cidr_addresses'] = ['10.164.0.30/32']
geo_postgresql['trust_auth_cidr_addresses'] = %w(10.164.0.34/32 10.164.0.35/32)
gitlab_rails['db_host'] = '10.164.0.30'
gitlab_rails['auto_migrate'] = false
alertmanager['enable'] = false
consul['enable'] = false
gitaly['enable'] = false
gitlab_monitor['enable'] = false
gitlab_workhorse['enable'] = false
nginx['enable'] = false
node_exporter['enable'] = false
pgbouncer_exporter['enable'] = false
postgresql['enable'] = false
prometheus['enable'] = false
redis['enable'] = false
redis_exporter['enable'] = false
repmgr['enable'] = false
sidekiq['enable'] = false
unicorn['enable'] = false
geo_secondary['auto_migrate'] = false

Happy to include if needed.

Edited by Fabian Zimmer