Skip to content

Cannot run pg tasks like backup because "pg_dump: server version: 12.1; pg_dump version: 11.7"

Summary

After upgrading of the external pg database from 9.6 to 12.x the backup utilities don't work anymore. They show:

pg_dump: server version: 12.1; pg_dump version: 11.7

Steps to reproduce

  1. Run gitlab as helm chart with pg sql 9.6 and gitlab helm chart 3.3.12
  2. Upgrade the external database from 9.6 to 12.1
  3. restart all sidekiq, unicorn, unicorn pods:
      kubectl delete pods -lapp=sidekiq,release=$APP_NAME
      kubectl delete pods -lapp=unicorn,release=$APP_NAME
      kubectl delete pods -lapp=task-runner,release=$APP_NAME
  1. try to run backup-utility in the task-runner:
Info
2020-08-16 09:19:59.178 CEST
Dumping PostgreSQL database gitlabhq_testing ... [FAILED]
Error
2020-08-16 09:19:59.154 CEST
(See full trace by running task with --trace)
Error
2020-08-16 09:19:59.154 CEST
Tasks: TOP => gitlab:backup:db:create
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/bin/bundle:3:in `<main>'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/bin/bundle:3:in `load'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/lib/tasks/gitlab/backup.rake:118:in `block (4 levels) in <top (required)>'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/lib/backup/database.rb:45:in `dump'
Error
2020-08-16 09:19:59.154 CEST
Backup::Error: Backup failed
Error
2020-08-16 09:19:59.154 CEST
rake aborted!
Error
2020-08-16 09:19:59.153 CEST
pg_dump: aborting because of server version mismatch

Configuration used

global:
  edition: ce
  hosts:
    domain: example.org
    https: true
    externalIP: 133.33.3.37
    ssh: ~
    gitlab:
       name: gitlab.example.org
    registry:
       name: registry.example.org

  ## doc/charts/globals.md#configure-ingress-settings
  ingress:
    configureCertmanager: true
    enabled: true
    tls:
      enabled: true

  ## doc/charts/globals.md#configure-postgresql-settings
  psql:
    password:
      secret: gitlab-pg
      key: password
    host: 133.33.33.37
    port: 5432
    username: gitlab
    database: gitlabhq_testing

gitlab:
  task-runner:
    backups:
      cron:
        enabled: true
        schedule: "20 0 * * *"
        extraArgs: "--skip lfs --skip packages --skip artifacts --skip uploads --skip registry --skip repositories"
      objectStorage:
        backend: gcs
        config:
          secret: google-application-credentials
          key: gcs-application-credentials-file
          gcpProject: example-gitlab-testing

Current behavior

pg_dump: server version: 12.1; pg_dump version: 11.7

and no backup happens!

Expected behavior

Backup works as expected.

Versions

  • Chart: 3.3.12
  • Platform:
    • Cloud: GKE
  • Kubernetes: (kubectl version)
    • Client: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
    • Server: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.42", GitCommit:"42bef28c2031a74fc68840fce56834ff7ea08518", GitTreeState:"clean", BuildDate:"2020-06-02T16:07:00Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}
  • Helm: (helm version)
    • Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
    • Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}

Relevant logs

Info
2020-08-16 09:19:59.178 CEST
Dumping PostgreSQL database gitlabhq_testing ... [FAILED]
Error
2020-08-16 09:19:59.154 CEST
(See full trace by running task with --trace)
Error
2020-08-16 09:19:59.154 CEST
Tasks: TOP => gitlab:backup:db:create
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/bin/bundle:3:in `<main>'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/bin/bundle:3:in `load'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/lib/tasks/gitlab/backup.rake:118:in `block (4 levels) in <top (required)>'
Error
2020-08-16 09:19:59.154 CEST
/srv/gitlab/lib/backup/database.rb:45:in `dump'
Error
2020-08-16 09:19:59.154 CEST
Backup::Error: Backup failed
Error
2020-08-16 09:19:59.154 CEST
rake aborted!
Error
2020-08-16 09:19:59.153 CEST
pg_dump: aborting because of server version mismatch
Edited by Jan Schütze