Skip to content

Cannot upgrade chart 2.6.9 to 3.3.13

Summary

After upgrading from helm chart 2.6.9 to helm chart 3.3.13 I get errors when restoring the backed up database.

Steps to reproduce

Configuration used

certmanager:
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
certmanager-issuer:
  email: SANITIZED
gitlab:
  gitaly:
    persistence:
      reclaimPolicy: Retain
  gitlab-shell:
    minReplicas: 1
  migrations:
    enabled: true
  sidekiq:
    minReplicas: 1
    resources:
      limits:
        memory: 1.5G
      requests:
        cpu: 50m
        memory: 625M
  task-runner:
    backups:
      objectStorage:
        config:
          key: config
          secret: SANITIZED
    enabled: true
  unicorn:
    minReplicas: 1
    resources:
      limits:
        memory: 1.5G
      requests:
        cpu: 100m
        memory: 900M
    workhorse:
      resources:
        limits:
          memory: 100M
        requests:
          cpu: 10m
          memory: 10M
gitlab-runner:
  install: false
global:
  appConfig:
    backups:
      bucket: SANITIZED
      objectStorage:
        backend: s3
      tmpBucket: SANITIZED
  edition: ce
  hosts:
    domain: SANITIZED
  ingress:
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: 0
      nginx.ingress.kubernetes.io/proxy-connect-timeout: 600
      nginx.ingress.kubernetes.io/proxy-read-timeout: 600
      nginx.ingress.kubernetes.io/proxy-send-timeout: 600
  smtp:
    address: SANITIZED
    authentication: login
    domain: SANITIZED
    enabled: true
    password:
      key: password
      secret: smtp-password
    port: 587
    starttls_auto: true
    user_name: SANITIZED
minio:
  persistence:
    reclaimPolicy: Retain
nginx-ingress:
  controller:
    minAvailable: 0
    replicaCount: 1
    resources:
      requests:
        cpu: 50m
        memory: 100Mi
  defaultBackend:
    minAvailable: 0
    replicaCount: 1
    resources:
      requests:
        cpu: 5m
        memory: 5Mi
postgresql:
  persistence:
    reclaimPolicy: Retain
prometheus:
  install: false
redis:
  persistence:
    reclaimPolicy: Retain
registry:
  hpa:
    minReplicas: 1

Current behavior

Database-upgrade script crashes with "undefined symbol: PQresultMemorySize". If I execute backup-utility directly on the gitlab-task-runner pod I get the same error.

If I start unicorn, sidekiq or gitlab-exporter (stopped by database-upgrade script) the dependencies container shows the same error.

Expected behavior

Database to be successfully restored.

Versions

  • Chart: 3.3.13
  • Platform:
    • Cloud: AWS
  • Kubernetes: (kubectl version)
    • Client: 1.16.2
    • Server: 1.16.9
  • Helm: (helm version)
    • Client: 2.16.12
    • Server: 2.16.12

Relevant logs

Executing database-upgrade from client machine

...
Unpacking backup
/srv/gitlab/vendor/bundle/ruby/2.6.0/bin/rake -f /srv/gitlab/Rakefile gitlab:db:drop_tables: symbol lookup error: /srv/gitlab/vendor/bundle/ruby/2.6.0/gems/pg-1.2.2/lib/pg_ext.so: undefined symbol: PQresultMemorySize
command terminated with exit code 127

Executing backup-utility directly on gitlab-task-runner

git@gitlab-task-runner-559b59c64c-r5x2t:/$ backup-utility --restore -t "database_upgrade" --skip repository,registry,uploads,artifa
cts,lfs,packages,repositories
Unpacking backup
/srv/gitlab/vendor/bundle/ruby/2.6.0/bin/rake -f /srv/gitlab/Rakefile gitlab:db:drop_tables: symbol lookup error: /srv/gitlab/vendo
r/bundle/ruby/2.6.0/gems/pg-1.2.2/lib/pg_ext.so: undefined symbol: PQresultMemorySize

Executing gitlab-rake on gitlab-task-runner

git@gitlab-task-runner-559b59c64c-r5x2t:/$ gitlab-rake db:migrate:status
/srv/gitlab/vendor/bundle/ruby/2.6.0/bin/rake -f /srv/gitlab/Rakefile db:migrate:status: symbol lookup error: /srv/gitlab/vendor/bu
ndle/ruby/2.6.0/gems/pg-1.2.2/lib/pg_ext.so: undefined symbol: PQresultMemorySize