HA with Geo upgrade from 12.0.9 to 12.1.9 does not upgrade PostgreSQL

Summary

The Geo team recently started conducting upgrade calls to a) understand the correct steps to upgrade an HA installation including Geo b) test upgrades between minor GitLab versions.

During the upgrade from 12.0.9 to 12.1.9 we expected the PostgreSQL database to be updated from 9.6.11 to 10.7; however, no PostgreSQL upgrade was performed. The resulting, healthy HA + Geo Configuration is still running PostgreSQL 9.6.11

It is unclear what the instructions are to actually upgrade PostgreSQL and/or if this is expected.

Steps to reproduce

We followed steps as described in !3577 (merged) These steps are new and we never had any official documentation for how to upgrade HA + Geo.

What is the current bug behavior?

We follow steps outlined in !3577 (merged) but PostgreSQL remains at version 9.6.11.

What is the expected correct behavior?

PostgresSQL should be updated to 10.7 unless sudo touch /etc/gitlab/disable-postgresql-upgrade is created. See https://docs.gitlab.com/ee/administration/geo/replication/version_specific_updates.html#updating-to-gitlab-121

Relevant logs

Details of package version

Environment details

We are using an HA cluster in the following configuration:

Primary (US)

VM Instance
ha-postgresql
ha-redis
ha-gitaly
ha-app-01
ha-app-02
ha-load-balancer

Secondary (EU)

VM Instance
geo-postgresql
geo-tracking
geo-redis
geo-gitaly
geo-app-01
geo-app-02
geo-load-balancer

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

PRIMARY HA APP01

external_url 'http://primary-ha.gogitlab.ml'
roles ['application_role']
nginx['enable'] = true
gitaly['enable'] = false
git_data_dirs({
  'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.142.0.11
:8075' }
})
gitlab_rails['gitaly_token'] = ''
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '10.142.0.9' # IP/hostname of database server
gitlab_rails['db_password'] = ''
gitlab_rails['redis_port'] = '6379'
gitlab_rails['redis_host'] = '10.142.0.10' # IP/hostname of Redis server
gitlab_rails['redis_password'] = ''

roles ['geo_primary_role']



PRIMARY HA APP02

external_url 'http://primary-ha.gogitlab.ml'
roles ['application_role']
nginx['enable'] = true
gitaly['enable'] = false
git_data_dirs({
  'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.142.0.11
:8075' }
})
gitlab_rails['gitaly_token'] = ''
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '10.142.0.9' # IP/hostname of database server
gitlab_rails['db_password'] = ''
gitlab_rails['redis_port'] = '6379'
gitlab_rails['redis_host'] = '10.142.0.10' # IP/hostname of Redis server
gitlab_rails['redis_password'] = ''
roles ['geo_primary_role']
gitlab_rails['auto_migrate'] = false


PRIMARY HA GITALY

external_url 'http://gitlab.example.com'
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
gitlab_rails['rake_cache_clear'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['internal_api_url'] = 'http://primary-ha.gogitlab.ml'
gitaly['listen_addr'] = "10.142.0.11:8075"
gitaly['auth_token'] = ''
gitaly['storage'] = [
  { 'name' => 'default', 'path' => '/var/opt/gitlab/git-data' },
]


PRIMARY HA PoSTGRESQL

external_url 'http://primary-ha-postgresql.gogitlab.ml'
roles ['geo_primary_role', 'postgres_role']
repmgr['enable'] = false
consul['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
pgbouncer_exporter['enable'] = false
redis_exporter['enable'] = false
gitlab_monitor['enable'] = false
postgresql['listen_address'] = '10.142.0.9'
postgresql['port'] = 5432
postgresql['sql_user_password'] = ''
postgresql['trust_auth_cidr_addresses'] = %w(10.142.0.12/32 10.142.0.13/32)
postgresql['md5_auth_cidr_addresses'] = ['10.142.0.9/32', '10.164.0.30/32']
gitlab_rails['auto_migrate'] = false
postgresql['max_replication_slots'] = 1


PRIMARY HA REDIS

external_url 'http://primary-ha-redis.gogitlab.ml'
redis['enable'] = true
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
unicorn['enable'] = false
postgresql['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
pgbouncer_exporter['enable'] = false
gitlab_monitor['enable'] = false
gitaly['enable'] = false
redis['bind'] = '10.142.0.10'
redis['port'] = 6379
redis['password'] = 'password'
gitlab_rails['auto_migrate'] = false


SECONDARY HA APP01

external_url 'http://secondary-ha.gogitlab.ml'
roles ['geo_secondary_role', 'application_role']
nginx['enable'] = true
gitlab_rails['auto_migrate'] = false
geo_secondary['db_host'] = '10.164.0.31'
geo_secondary['db_password'] = ''
geo_postgresql['enable'] = false
gitlab_rails['db_host'] = '10.164.0.30'
gitlab_rails['db_password'] = ''
gitlab_rails['redis_host'] = '10.164.0.32'
gitlab_rails['redis_password'] = ''
gitaly['enable'] = false
git_data_dirs({
  'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.164.0.33
:8075' }
})
gitlab_rails['gitaly_token'] = ''


SECONDARY HA APP02

external_url 'http://secondary-ha.gogitlab.ml'
roles ['geo_secondary_role', 'application_role']
gitlab_rails['auto_migrate'] = false
nginx['enable'] = true
geo_secondary['db_host'] = '10.164.0.31'
geo_secondary['db_password'] = ''
geo_postgresql['enable'] = false
gitlab_rails['db_host'] = '10.164.0.30'
gitlab_rails['db_password'] = ''
gitlab_rails['redis_host'] = '10.164.0.32'
gitlab_rails['redis_password'] = ''
gitaly['enable'] = false
git_data_dirs({
  'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.164.0.33
:8075' }
})
gitlab_rails['gitaly_token'] = ''
geo_secondary['auto_migrate'] = false


SECONDARY HA GITALY

external_url 'http://gitlab.example.com'
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
gitlab_rails['rake_cache_clear'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['internal_api_url'] = 'http://secondary-ha.gogitlab.ml'
gitaly['listen_addr'] = "10.164.0.33:8075"
gitaly['auth_token'] = ''
gitaly['storage'] = [
  { 'name' => 'default', 'path' => '/var/opt/gitlab/git-data' },
]
geo_secondary['auto_migrate'] = false


SECONDARY HA POSTGRESQL

external_url 'http://secondary-ha-postgresql.gogitlab.ml'
roles ['postgres_role']
repmgr['enable'] = false
postgresql['listen_address'] = '10.164.0.30'
postgresql['md5_auth_cidr_addresses'] = ['10.164.0.30/32', '10.164.0.31/32', '10.164.0.34/32'
, '10.164.0.35/32']
postgresql['hot_standby'] = 'on'
postgresql['sql_user_password'] = ''
gitlab_rails['db_password'] = ''
geo_postgresql['enable'] = false
gitlab_rails['auto_migrate'] = false
geo_secondary['auto_migrate'] = false


SECONDARY HA REDIS

external_url 'http://secondary-ha-redis.gogitlab.ml'
redis['enable'] = true
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
unicorn['enable'] = false
postgresql['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
pgbouncer_exporter['enable'] = false
gitlab_monitor['enable'] = false
gitaly['enable'] = false
redis['bind'] = '10.164.0.32'
redis['port'] = 6379
redis['password'] = ''
gitlab_rails['auto_migrate'] = false
geo_secondary['auto_migrate'] = false


SECONDARY HA TRACKING

external_url 'http://secondary-ha-tracking.gogitlab.ml'
geo_postgresql['enable'] = true
geo_postgresql['listen_address'] = '10.164.0.31'
geo_postgresql['sql_user_password'] = ''
geo_secondary['db_fdw'] = true
geo_postgresql['fdw_external_password'] = ''
geo_postgresql['md5_auth_cidr_addresses'] = ['10.164.0.30/32']
geo_postgresql['trust_auth_cidr_addresses'] = %w(10.164.0.34/32 10.164.0.35/32)
gitlab_rails['db_host'] = '10.164.0.30'
gitlab_rails['auto_migrate'] = false
alertmanager['enable'] = false
consul['enable'] = false
gitaly['enable'] = false
gitlab_monitor['enable'] = false
gitlab_workhorse['enable'] = false
nginx['enable'] = false
node_exporter['enable'] = false
pgbouncer_exporter['enable'] = false
postgresql['enable'] = false
prometheus['enable'] = false
redis['enable'] = false
redis_exporter['enable'] = false
repmgr['enable'] = false
sidekiq['enable'] = false
unicorn['enable'] = false
geo_secondary['auto_migrate'] = false

Happy to include if needed.