When running Gitlab-EE HA 11.11.X and manually upgrading postgres, pg-dump is the incorrect version
Summary
After upgrading to Gitlab-EE 11.11.3 from 11.7.0, we ran the manual postgreSQL Upgrade to 10.7. After the upgrade completed, pg-dump no longer functions from a Gitlab 'Application server' as it is the incorrect version. We are currently running in a non-containerized HA Architecture:
- 2 Gitlab App Servers (backup rake task is running from one of these nodes)
- 3 Redis/Sentinel
- 3 Postgresql/repmgr
Steps to reproduce
- Upgrade Gitlab to 11.11.3
- Deploy manual PostgreSQL upgrade to 10.7
- Attempt to run
/opt/gitlab/bin/gitlab-rake gitlab:backup:create
from a gitlab application server
What is the current bug behavior?
The gitlab backup rake tasks fail with an error (See Below), this will actually cause Gitlab to throw 500 errors for a short period of time.
What is the expected correct behavior?
Gitlab should successfully dump the contents of the postgres databases for backup
Relevant logs
sudo /opt/gitlab/bin/gitlab-rake gitlab:backup:create
2019-06-18 09:53:43 -0400 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... pg_dump: server version: 10.7; pg_dump version: 9.6.11
pg_dump: aborting because of server version mismatch
[FAILED]
rake aborted!
Backup::Error: Backup failed
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/database.rb:50:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:99:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:10:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:db:create
(See full trace by running task with --trace)
Details of package version
Provide the package version installation details
gitlab-ee-11.11.3-ee.0.el7.x86_64
Environment details
- Operating System: RHEL 7.6
- Installation Target, remove incorrect values:
- VM: On-Prem Virtual Machine (VMWare)
- Installation Type, remove incorrect values:
- Upgrade from version gitlab-ee-11.7.0-ee.0.el7.x86_64
- Is there any other software running on the machine: No
- Is this a single or multiple node installation? Multiple
- 2 Gitlab App Servers
- 3 Redis/Sentinel
- 3 Postgresql/repmgr
- Resources
- CPU: 2 vCPU
- Memory total: 4GB Memory
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
N/A
Edited by Michael G. Gerhart