@ljlane@fzimmer@mendeni Added this issue for verifying upgrading a Geo install to PG12. For now, I mostly copied the description in #4975 (closed). I believe we will only need to test the upgrade from PG11, but let me know if I'm mistaken.
Perfect, thanks for getting this created @nhxnguyen. You are correct about only needing to test upgrades from PG11 to PG12 since we have guardrails in place to ensure users upgrade to 13.0 before 13.x.
Customer using Gitlab EE Premium with GEO and Postgresql:
We need this resolved before we are required to update our database. We use GEO and cannot update our databases until this is supported. If this is not supported by November 2020, we will be forced to stay on Gitlab 13.5.0 until Postgresql 12 is supported on GEO?
Hi @atbeasley, the information in this error message is outdated. There was an MR open to fix it, but we ended up removing that notice entirely: gitlab!38225 (merged)
The fix was included in %13.4, and picked into %13.3 last week.
Sorry if I'm missing something here. Do we have a new date for when Postgres 12 will be required? We are still on 11.7 at the moment, being users of Geo.
@mnielsen We have an issue on our board to update PostgreSQL to 12. It sounds like this will happen automatically with 13.6, though. Can you confirm? We are using Geo with an Omnibus installation and we grab the latest package with yum weekly.
In %14.0 PG 12 becomes the minimum required version
So yes, the current plan is for this happen automatically in GitLab 13.6. It won't be until GitLab 14.0 that PostgreSQL 12 will be the minimum required version.
/cc @ljlane in case any of this is incorrect/incomplete. Thanks!
@christo.alexander_cf@mnielsen we will only be auto-upgrading non-HA and non-geo omnibus installs. This is due to the upgrades being destructive to data on secondaries, not something we want to spring on users.
I just updated the epic that @mnielsen linked. We won't be defaulting to PostgreSQL 12 until %13.7 now. We decided to push it out by one milestone to allow for some extra Geo and HA testing.
@ljlane not much, adding it probably is like 10min of work, though testing it and getting it ready for review might take another hour or two.
(This is for adding a change that stops and throws a message when users are running PG 11, and repmgr, and are trying to run pg-upgrade, letting them know they need to migrate to patroni first)
Single primary node, secondary db and app node, standalone tracking db node
Upgraded to PG12 on primary and secondary nodes following upgrade instructions and re-initiated replication.
Ran into an error with gitlab-ctl pg-upgrade on the tracking db node because /var/opt/gitlab/postgresql/data did not exist. After creating the directory, upgrade was successful.
Error with pg-upgrade when /var/opt/gitlab/postgresql/data/ directory doesn't exist
root@nn-pg12-upgrade-ha-secondary-geo-1:~# gitlab-ctl pg-upgrade -V 12Checking for an omnibus managed postgresql: OKChecking if postgresql['version'] is set: OKChecking if we already upgraded: NOT OKChecking for a newer version of PostgreSQL to installUpgrading PostgreSQL to 12.4Checking if disk for directory /var/opt/gitlab/postgresql/data has enough free space for PostgreSQL upgrade:Traceback (most recent call last): 14: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `' 13: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load' 12: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `' 11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run' 10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category' 9: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:161:in `block in load_file' 8: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:161:in `each' 7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:162:in `block (2 levels) in load_file' 6: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:122:in `progress_message' 5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:165:in `block (3 levels) in load_file' 4: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:53:in `enough_free_space?' 3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:57:in `space_needed' 2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:14:in `get_command_output' 1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.1.6/lib/mixlib/shellout.rb:287:in `error!'/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.1.6/lib/mixlib/shellout.rb:300:in `invalid!': Expected process to exit with [0], but received '1' (Mixlib::ShellOut::ShellCommandFailed)---- Begin output of du -s --block-size=1m /var/opt/gitlab/postgresql/data ----STDOUT:STDERR: du: cannot access '/var/opt/gitlab/postgresql/data': No such file or directory---- End output of du -s --block-size=1m /var/opt/gitlab/postgresql/data ----Ran du -s --block-size=1m /var/opt/gitlab/postgresql/data returned 1 13: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `' 12: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load' 11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `' 10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run' 9: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category' 8: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:161:in `block in load_file' 7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:161:in `each' 6: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:162:in `block (2 levels) in load_file' 5: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:122:in `progress_message' 4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:165:in `block (3 levels) in load_file' 3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:53:in `enough_free_space?' 2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:57:in `space_needed' 1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:12:in `get_command_output'/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:16:in `rescue in get_command_output': GitlabCtl::Errors::ExecutionError (GitlabCtl::Errors::ExecutionError)
Ran into an error with gitlab-ctl pg-upgrade on the tracking db node because /var/opt/gitlab/postgresql/data did not exist. After creating the directory, upgrade was successful.
@mkozono@cat@abuerer Is this a known issue? I tried searching through some other PG upgrade issues such as #4975 (comment 321596835) and didn't see anything. So I'm wondering if this was just somehow specific to my setup.
I've not seen that before, I think it's new since the free space check was added in 13.3? All I could find was #5660 (closed) which is related (as in, it's another bug with the free space check but not the same issue), sounds like we may need an issue for it
@nhxnguyen Was this a fresh install? I think the upgrade will fail if the directory does not exist because it takes a backup of the directory before upgrading the data.
It was a fresh install and failed during the free space check. I didn't realize it was just added in 13.3, so that would explain why we're just running into this. Opened #5664 (closed) as a follow-up.
Nick Nguyenmarked the checklist item Single node primary and secondaries as completed
marked the checklist item Single node primary and secondaries as completed
Nick Nguyenmarked the checklist item Separate database nodes on primary. Secondary with separate node running postgresql and geo-postgresql. as completed
marked the checklist item Separate database nodes on primary. Secondary with separate node running postgresql and geo-postgresql. as completed
Additional Note: I realized that the tests I was performing in this assumed that the upgrade would be done on the originally installed GitLab version, which in this case already had Geo support for PostgreSQL 12. Since versions prior to GitLab 13.4 would not create the gitlab-geo.conf file, I wanted to verify that upgrading from one of these versions, then upgrading from PG 11 to 12 would work.
I tested an upgrade from GitLab 13.3.9 (with PG 11) to the latest GitLab 13.5 on both the primary and secondary nodes. Then I upgraded from PG 11 to 12 using the same steps as previous upgrade tests. The upgrade was successful.