pg-upgrade fails on free space check for standalone Geo tracking node
Summary
In the process of testing upgrading from PG11 to PG12 on a standalone Geo tracking database node, I encountered an issue during the free space check. It is looking for the PostgreSQL data_dir
but that directory does not exist since the node is enabled only for geo-postgresql
. The default data_dir
for that service is /var/opt/gitlab/geo-postgresql/data
.
Steps to reproduce
1.Set up a Geo deployment with standalone tracking database using PostgreSQL 11.
- Upgrade the Geo primary database and main read-only replica to PG12 with
gitlab-ctl pg-upgrade -V 12
- Attempt to upgrade the Geo tracking database to PG12 with
gitlab-ctl pg-upgrade -V 12
What is the current bug behavior?
The step Checking if disk for directory /var/opt/gitlab/postgresql/data has enough free space for PostgreSQL upgrade
fails with STDERR: du: cannot access '/var/opt/gitlab/postgresql/data': No such file or directory
What is the expected correct behavior?
It should run the check on the geo-postgresql
data directory.
Relevant logs
Relevant logs
root@nn-pg12-upgrade-ha-secondary-geo-1:~# gitlab-ctl pg-upgrade -V 12
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 12.4
Checking if disk for directory /var/opt/gitlab/postgresql/data has enough free space for PostgreSQL upgrade:Traceback (most recent call last):
14: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `'
13: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
12: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `'
11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
9: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:161:in `block in load_file'
8: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:161:in `each'
7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:162:in `block (2 levels) in load_file'
6: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:122:in `progress_message'
5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:165:in `block (3 levels) in load_file'
4: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:53:in `enough_free_space?'
3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:57:in `space_needed'
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:14:in `get_command_output'
1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.1.6/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.1.6/lib/mixlib/shellout.rb:300:in `invalid!': Expected process to exit with [0], but received '1' (Mixlib::ShellOut::ShellCommandFailed)
---- Begin output of du -s --block-size=1m /var/opt/gitlab/postgresql/data ----
STDOUT:
STDERR: du: cannot access '/var/opt/gitlab/postgresql/data': No such file or directory
---- End output of du -s --block-size=1m /var/opt/gitlab/postgresql/data ----
Ran du -s --block-size=1m /var/opt/gitlab/postgresql/data returned 1
13: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `'
12: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `'
10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
9: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
8: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:161:in `block in load_file'
7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:161:in `each'
6: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:162:in `block (2 levels) in load_file'
5: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:122:in `progress_message'
4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:165:in `block (3 levels) in load_file'
3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:53:in `enough_free_space?'
2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:57:in `space_needed'
1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:12:in `get_command_output'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:16:in `rescue in get_command_output': GitlabCtl::Errors::ExecutionError (GitlabCtl::Errors::ExecutionError)
Details of package version
Tested with gitlab-ee=13.3.6+rnightly.169516.d5209202-0
Environment details
- Operating System:
REPLACE-WITH-DETAILS
- Installation Target, remove incorrect values:
- Bare Metal Machine
- VM: Digital Ocean, AWS, GCP, Azure, Other
REPLACE-WITH-DETAILS
- Other:
REPLACE-WITH-DETAILS
- Installation Type, remove incorrect values:
- New Installation
- Upgrade from version
REPLACE-WITH-DETAILS
- Other:
REPLACE-WITH-DETAILS
- Is there any other software running on the machine:
REPLACE-WITH-DETAILS
- Is this a single or multiple node installation?
- Resources
- CPU:
REPLACE-WITH-DETAILS
- Memory total:
REPLACE-WITH-DETAILS
- CPU: