<!--- Please read this! Before you create a new issue, please make sure that you already ran `sudo gitlab-ctl reconfigure` Also make sure to search in https://gitlab.com/gitlab-org/omnibus-gitlab/issues, to verify the issue you're about to submit isn't a duplicate. ---> ### Summary <!-- Describe in short the bug you've encountered --> The Geo team recently started conducting upgrade calls to a) understand the correct steps to upgrade an HA installation including Geo b) test upgrades between minor GitLab versions. During the upgrade from 12.0.9 to 12.1.9 we expected the PostgreSQL database to be updated from 9.6.11 to 10.7; however, no PostgreSQL upgrade was performed. The resulting, healthy HA + Geo Configuration is still running PostgreSQL 9.6.11 It is unclear what the instructions are to actually upgrade PostgreSQL and/or if this is expected. ### Steps to reproduce <!-- How one can reproduce the issue - this is very important --> We followed steps as described in https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/3577 These steps are new and we never had any official documentation for how to upgrade HA + Geo. ### What is the current *bug* behavior? <!-- What actually happens --> We follow steps outlined in https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/3577 but PostgreSQL remains at version 9.6.11. ### What is the expected *correct* behavior? <!-- What you should see instead --> PostgresSQL should be updated to 10.7 unless `sudo touch /etc/gitlab/disable-postgresql-upgrade` is created. See https://docs.gitlab.com/ee/administration/geo/replication/version_specific_updates.html#updating-to-gitlab-121 ### Relevant logs ### Details of package version ### Environment details We are using an HA cluster in the following configuration: #### Primary (US) | VM Instance | |-------------| | ha-postgresql| | ha-redis| | ha-gitaly| | ha-app-01| | ha-app-02| | ha-load-balancer| #### Secondary (EU) | VM Instance | |-------------| | geo-postgresql | | geo-tracking | | geo-redis | | geo-gitaly | | geo-app-01 | | geo-app-02 | | geo-load-balancer | ### Configuration details <details> <summary> Provide the relevant sections of `/etc/gitlab/gitlab.rb` </summary> <pre> <!-- Paste the details but omit lines starting with `#`. Relevant lines can be retrieved by running `grep -v -e '^#' -e '^$' /etc/gitlab/gitlab.rb`. Be certain to sanitize sensitive configuration such as passwords and secrets. --> PRIMARY HA APP01 ``` external_url 'http://primary-ha.gogitlab.ml' roles ['application_role'] nginx['enable'] = true gitaly['enable'] = false git_data_dirs({ 'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.142.0.11 :8075' } }) gitlab_rails['gitaly_token'] = '' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' gitlab_rails['db_host'] = '10.142.0.9' # IP/hostname of database server gitlab_rails['db_password'] = '' gitlab_rails['redis_port'] = '6379' gitlab_rails['redis_host'] = '10.142.0.10' # IP/hostname of Redis server gitlab_rails['redis_password'] = '' roles ['geo_primary_role'] ``` PRIMARY HA APP02 ``` external_url 'http://primary-ha.gogitlab.ml' roles ['application_role'] nginx['enable'] = true gitaly['enable'] = false git_data_dirs({ 'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.142.0.11 :8075' } }) gitlab_rails['gitaly_token'] = '' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' gitlab_rails['db_host'] = '10.142.0.9' # IP/hostname of database server gitlab_rails['db_password'] = '' gitlab_rails['redis_port'] = '6379' gitlab_rails['redis_host'] = '10.142.0.10' # IP/hostname of Redis server gitlab_rails['redis_password'] = '' roles ['geo_primary_role'] gitlab_rails['auto_migrate'] = false ``` PRIMARY HA GITALY ``` external_url 'http://gitlab.example.com' postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false prometheus['enable'] = false unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false gitlab_rails['internal_api_url'] = 'http://primary-ha.gogitlab.ml' gitaly['listen_addr'] = "10.142.0.11:8075" gitaly['auth_token'] = '' gitaly['storage'] = [ { 'name' => 'default', 'path' => '/var/opt/gitlab/git-data' }, ] ``` PRIMARY HA PoSTGRESQL ``` external_url 'http://primary-ha-postgresql.gogitlab.ml' roles ['geo_primary_role', 'postgres_role'] repmgr['enable'] = false consul['enable'] = false prometheus['enable'] = false alertmanager['enable'] = false pgbouncer_exporter['enable'] = false redis_exporter['enable'] = false gitlab_monitor['enable'] = false postgresql['listen_address'] = '10.142.0.9' postgresql['port'] = 5432 postgresql['sql_user_password'] = '' postgresql['trust_auth_cidr_addresses'] = %w(10.142.0.12/32 10.142.0.13/32) postgresql['md5_auth_cidr_addresses'] = ['10.142.0.9/32', '10.164.0.30/32'] gitlab_rails['auto_migrate'] = false postgresql['max_replication_slots'] = 1 ``` PRIMARY HA REDIS ``` external_url 'http://primary-ha-redis.gogitlab.ml' redis['enable'] = true sidekiq['enable'] = false gitlab_workhorse['enable'] = false unicorn['enable'] = false postgresql['enable'] = false nginx['enable'] = false prometheus['enable'] = false alertmanager['enable'] = false pgbouncer_exporter['enable'] = false gitlab_monitor['enable'] = false gitaly['enable'] = false redis['bind'] = '10.142.0.10' redis['port'] = 6379 redis['password'] = 'password' gitlab_rails['auto_migrate'] = false ``` SECONDARY HA APP01 ``` external_url 'http://secondary-ha.gogitlab.ml' roles ['geo_secondary_role', 'application_role'] nginx['enable'] = true gitlab_rails['auto_migrate'] = false geo_secondary['db_host'] = '10.164.0.31' geo_secondary['db_password'] = '' geo_postgresql['enable'] = false gitlab_rails['db_host'] = '10.164.0.30' gitlab_rails['db_password'] = '' gitlab_rails['redis_host'] = '10.164.0.32' gitlab_rails['redis_password'] = '' gitaly['enable'] = false git_data_dirs({ 'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.164.0.33 :8075' } }) gitlab_rails['gitaly_token'] = '' ``` SECONDARY HA APP02 ``` external_url 'http://secondary-ha.gogitlab.ml' roles ['geo_secondary_role', 'application_role'] gitlab_rails['auto_migrate'] = false nginx['enable'] = true geo_secondary['db_host'] = '10.164.0.31' geo_secondary['db_password'] = '' geo_postgresql['enable'] = false gitlab_rails['db_host'] = '10.164.0.30' gitlab_rails['db_password'] = '' gitlab_rails['redis_host'] = '10.164.0.32' gitlab_rails['redis_password'] = '' gitaly['enable'] = false git_data_dirs({ 'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.164.0.33 :8075' } }) gitlab_rails['gitaly_token'] = '' geo_secondary['auto_migrate'] = false ``` SECONDARY HA GITALY ``` external_url 'http://gitlab.example.com' postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false prometheus['enable'] = false unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false gitlab_rails['internal_api_url'] = 'http://secondary-ha.gogitlab.ml' gitaly['listen_addr'] = "10.164.0.33:8075" gitaly['auth_token'] = '' gitaly['storage'] = [ { 'name' => 'default', 'path' => '/var/opt/gitlab/git-data' }, ] geo_secondary['auto_migrate'] = false ``` SECONDARY HA POSTGRESQL ``` external_url 'http://secondary-ha-postgresql.gogitlab.ml' roles ['postgres_role'] repmgr['enable'] = false postgresql['listen_address'] = '10.164.0.30' postgresql['md5_auth_cidr_addresses'] = ['10.164.0.30/32', '10.164.0.31/32', '10.164.0.34/32' , '10.164.0.35/32'] postgresql['hot_standby'] = 'on' postgresql['sql_user_password'] = '' gitlab_rails['db_password'] = '' geo_postgresql['enable'] = false gitlab_rails['auto_migrate'] = false geo_secondary['auto_migrate'] = false ``` SECONDARY HA REDIS ``` external_url 'http://secondary-ha-redis.gogitlab.ml' redis['enable'] = true sidekiq['enable'] = false gitlab_workhorse['enable'] = false unicorn['enable'] = false postgresql['enable'] = false nginx['enable'] = false prometheus['enable'] = false alertmanager['enable'] = false pgbouncer_exporter['enable'] = false gitlab_monitor['enable'] = false gitaly['enable'] = false redis['bind'] = '10.164.0.32' redis['port'] = 6379 redis['password'] = '' gitlab_rails['auto_migrate'] = false geo_secondary['auto_migrate'] = false ``` SECONDARY HA TRACKING ``` external_url 'http://secondary-ha-tracking.gogitlab.ml' geo_postgresql['enable'] = true geo_postgresql['listen_address'] = '10.164.0.31' geo_postgresql['sql_user_password'] = '' geo_secondary['db_fdw'] = true geo_postgresql['fdw_external_password'] = '' geo_postgresql['md5_auth_cidr_addresses'] = ['10.164.0.30/32'] geo_postgresql['trust_auth_cidr_addresses'] = %w(10.164.0.34/32 10.164.0.35/32) gitlab_rails['db_host'] = '10.164.0.30' gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false consul['enable'] = false gitaly['enable'] = false gitlab_monitor['enable'] = false gitlab_workhorse['enable'] = false nginx['enable'] = false node_exporter['enable'] = false pgbouncer_exporter['enable'] = false postgresql['enable'] = false prometheus['enable'] = false redis['enable'] = false redis_exporter['enable'] = false repmgr['enable'] = false sidekiq['enable'] = false unicorn['enable'] = false geo_secondary['auto_migrate'] = false ``` </pre> </details> Happy to include if needed.