Cannot run pg tasks like backup because "pg_dump: server version: 12.1; pg_dump version: 11.7"
Summary
After upgrading of the external pg database from 9.6 to 12.x the backup utilities don't work anymore. They show:
pg_dump: server version: 12.1; pg_dump version: 11.7
Steps to reproduce
- Run gitlab as helm chart with pg sql 9.6 and gitlab helm chart
3.3.12
- Upgrade the external database from 9.6 to 12.1
- restart all sidekiq, unicorn, unicorn pods:
kubectl delete pods -lapp=sidekiq,release=$APP_NAME
kubectl delete pods -lapp=unicorn,release=$APP_NAME
kubectl delete pods -lapp=task-runner,release=$APP_NAME
- try to run backup-utility in the task-runner:
Info
2020-08-16 09:19:59.178 CEST
Dumping PostgreSQL database gitlabhq_testing ... [FAILED]
Error
2020-08-16 09:19:59.154 CEST
(See full trace by running task with --trace)
Error
2020-08-16 09:19:59.154 CEST
Tasks: TOP => gitlab:backup:db:create
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/bin/bundle:3:in `<main>'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/bin/bundle:3:in `load'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/lib/tasks/gitlab/backup.rake:118:in `block (4 levels) in <top (required)>'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/lib/backup/database.rb:45:in `dump'
Error
2020-08-16 09:19:59.154 CEST
Backup::Error: Backup failed
Error
2020-08-16 09:19:59.154 CEST
rake aborted!
Error
2020-08-16 09:19:59.153 CEST
pg_dump: aborting because of server version mismatch
Configuration used
global:
edition: ce
hosts:
domain: example.org
https: true
externalIP: 133.33.3.37
ssh: ~
gitlab:
name: gitlab.example.org
registry:
name: registry.example.org
## doc/charts/globals.md#configure-ingress-settings
ingress:
configureCertmanager: true
enabled: true
tls:
enabled: true
## doc/charts/globals.md#configure-postgresql-settings
psql:
password:
secret: gitlab-pg
key: password
host: 133.33.33.37
port: 5432
username: gitlab
database: gitlabhq_testing
gitlab:
task-runner:
backups:
cron:
enabled: true
schedule: "20 0 * * *"
extraArgs: "--skip lfs --skip packages --skip artifacts --skip uploads --skip registry --skip repositories"
objectStorage:
backend: gcs
config:
secret: google-application-credentials
key: gcs-application-credentials-file
gcpProject: example-gitlab-testing
Current behavior
pg_dump: server version: 12.1; pg_dump version: 11.7
and no backup happens!
Expected behavior
Backup works as expected.
Versions
- Chart: 3.3.12
- Platform:
- Cloud: GKE
- Kubernetes: (
kubectl version
)- Client: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
- Server: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.42", GitCommit:"42bef28c2031a74fc68840fce56834ff7ea08518", GitTreeState:"clean", BuildDate:"2020-06-02T16:07:00Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}
- Helm: (
helm version
)- Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
- Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Relevant logs
Info
2020-08-16 09:19:59.178 CEST
Dumping PostgreSQL database gitlabhq_testing ... [FAILED]
Error
2020-08-16 09:19:59.154 CEST
(See full trace by running task with --trace)
Error
2020-08-16 09:19:59.154 CEST
Tasks: TOP => gitlab:backup:db:create
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/bin/bundle:3:in `<main>'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/bin/bundle:3:in `load'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/lib/tasks/gitlab/backup.rake:118:in `block (4 levels) in <top (required)>'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/lib/backup/database.rb:45:in `dump'
Error
2020-08-16 09:19:59.154 CEST
Backup::Error: Backup failed
Error
2020-08-16 09:19:59.154 CEST
rake aborted!
Error
2020-08-16 09:19:59.153 CEST
pg_dump: aborting because of server version mismatch
Edited by Jan Schütze