`pg-upgrade` fails to upgrade a standalone Geo tracking DB
mkozono-ansible@mkozono-omnibus4838b-secondary-geo:~$ sudo gitlab-ctl pg-upgrade --target-version=11
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 11.7
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details
Please hit Ctrl-C now if you want to cancel the operation.
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: crond: 0s, normally up
ok: down: grafana: 0s, normally up
ok: down: logrotate: 0s, normally up
Toggling services: OK
There was an error fetching locale and encoding information from the database
Please ensure the database is running and functional before running pg-upgrade
STDOUT:
STDERR: psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
== Fatal error ==
Please check error logs
== Reverting ==
Traceback (most recent call last):
13: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
12: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
9: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
8: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:189:in `block in load_file'
7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:244:in `general_upgrade'
6: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:196:in `common_pre_upgrade'
5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:338:in `get_locale_encoding'
4: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:78:in `fetch_lc_ctype'
3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:66:in `run_query'
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:14:in `get_command_output'
1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:300:in `invalid!': Expected process to exit with [0], but received '2' (Mixlib::ShellOut::ShellCommandFailed)
---- Begin output of gitlab-psql -d postgres -c 'SHOW LC_CTYPE' -q -t ----
STDOUT:
STDERR: psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
---- End output of gitlab-psql -d postgres -c 'SHOW LC_CTYPE' -q -t ----
Ran gitlab-psql -d postgres -c 'SHOW LC_CTYPE' -q -t returned 2
12: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
11: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
9: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:189:in `block in load_file'
6: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:244:in `general_upgrade'
5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:196:in `common_pre_upgrade'
4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:338:in `get_locale_encoding'
3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:78:in `fetch_lc_ctype'
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:66:in `run_query'
1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:12:in `get_command_output'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:16:in `rescue in get_command_output': GitlabCtl::Errors::ExecutionError (GitlabCtl::Errors::ExecutionError)
14: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
13: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
12: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
9: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:189:in `block in load_file'
8: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:244:in `general_upgrade'
7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:196:in `common_pre_upgrade'
6: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:337:in `get_locale_encoding'
5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:346:in `rescue in get_locale_encoding'
4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:525:in `die'
3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:487:in `revert'
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:46:in `tmp_data_dir'
1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:40:in `data_dir'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:40:in `realpath': No such file or directory @ realpath_rec - /var/opt/gitlab/postgresql (Errno::ENOENT)
Looking for a fix
- It appears pg-upgrade assumes the tracking DB has the
geo-secondary
role, which is not the case: https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/12.9.2+ee.0/files/gitlab-ctl-commands/pg-upgrade.rb#L184- Therefore it did not detect a Geo secondary, and attempted to run
general_upgrade
- We don’t want the
geo-secondary
role on the tracking DB because it does a bunch of things that we explicitly don’t want when the tracking DB stands alone. We just enable thegeo-postgresql
service.
- Therefore it did not detect a Geo secondary, and attempted to run
- I patched the condition
elsif @roles.include?('geo-secondary')
=>elsif @roles.include?('geo-secondary') || service_enabled?('geo-postgresql')
and reranpg-upgrade
. I got another error:
<snip>
Initializing the new database: OK
Upgrading the data:Error upgrading the data to version 11.7
STDOUT:
could not write to log file "pg_upgrade_internal.log"
Failure, exiting
STDERR:
Upgrading the data: NOT OK
Traceback (most recent call last):
10: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
9: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
7: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:187:in `block in load_file'
4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:329:in `geo_secondary_upgrade'
3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:130:in `run_pg_upgrade'
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:130:in `raise'
1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:130:in `exception'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl.rb:27:in `initialize': wrong number of arguments (given 1, expected 3) (ArgumentError)
- And now it appears the PG server is on 11.7 but the data is on 10, so the server can't run.
2020-04-10_08:30:11.87492 FATAL: database files are incompatible with server
2020-04-10_08:30:11.87495 DETAIL: The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 11.7.
mkozono-ansible@mkozono-omnibus4838b-secondary-geo:~$ sudo gitlab-ctl pg-upgrade --target-version=11
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: OK
The latest version 11.7 is already running, nothing to do
- And
revert-pg-upgrade
fails from here:
mkozono-ansible@mkozono-omnibus4838b-secondary-geo:~$ sudo gitlab-ctl revert-pg-upgrade
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: crond: 3111s, normally up
ok: down: grafana: 3110s, normally up
ok: down: logrotate: 3110s, normally up
Toggling services: OK
Checking if we need to downgrade: NOT OK
Traceback (most recent call last):
8: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
7: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
5: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
4: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:48:in `block in load_file'
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:46:in `tmp_data_dir'
1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:40:in `data_dir'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:40:in `realpath': No such file or directory @ realpath_rec - /var/opt/gitlab/postgresql (Errno::ENOENT)
Setting ~S2 for now because I don't know of a workaround.
Edited by Michael Kozono