Skip to content

Cannot restore to helm-installed version.

Summary

When attempting to restore to a new installation of gitlab (chart version 4.6.3) from an Omnibus installation, the process fails with a series of postgres-related errors.

Steps to reproduce

Backup an omnibus installation, copy the backup to the s3 (minio) bucket for gitlab-backups for the helm chart, and attempt a normal restore.

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

certmanager:
  install: false
gitlab:
  task-runner:
    backups:
      objectStorage:
        config:
          secret: [redacted]
          key: config
  webservice:
    ingress:
      tls:
        secretName: release-gitlab-tls
gitlab-runner:
  install: false
global:
  appConfig:
    objectStorage:
      config:
        enabled: true
        direct_upload: false
        background_upload: true
    artifacts:
      enabled: true
      bucket: gitlab-artifacts-storage
      connection:
        secret: [redacted]
        key: connection
    backups:
      bucket: gitlab-backup-storage
      tmpBucket: gitlab-tmp-storage
    externalDiffs:
      enabled: true
      bucket: gitlab-externaldiffs-storage
      connection:
        secret: [redacted]
        key: connection
    lfs:
      enabled: true
      bucket: gitlab-lfs-storage
      connection:
        secret: [redacted]
        key: connection
    omniauth:
      enabled: true
      allowSingleSignOn: true
      blockAutoCreatedUsers: false
      autoLinkSamlUser: true
      autoLinkLdapUser: true
      providers:
        - secret: [redacted]
          key: provider
    packages:
      enabled: true
      bucket: gitlab-packages-storage
      connection:
        secret: [redacted]
        key: connection
    pseudonymizer:
      enabled: true
      bucket: gitlab-pseudonymizer-storage
      connection:
        secret: [redacted]
        key: connection
    uploads:
      enabled: true
      bucket: gitlab-uploads-storage
      connection:
        secret: [redacted]
        key: connection
  email:
    display_name: [redacted]
    from: [redacted]
    reply_to: [redacted]
  hosts:
    domain: [redacted]
    gitlab:
      name: [redacted]
    registry:
      name: [redacted]
  ingress:
    annotations: 
      cert-manager.io/cluster-issuer: letsencrypt-prod
      externalDNS: true
      external-dns.alpha.kubernetes.io/ttl: 300
      kubernetes.io/tls-acme: true
    class: nginx
    configureCertmanager: false
  minio:
    enabled: false
  psql:
    database: gitlab
    host: [redacted]
    password:
      secret: [redacted]
      key: postgresql-password
    username: [redacted]
  registry:
    bucket: gitlab-registry-storage
  smtp:
    address: [redacted]
    authentication: ""
    enabled: true
    port: 25
    tls: none
minio:
  ingress:
    tls:
      secretName: [redacted]
nginx-ingress:
  enabled: false
postgresql:
  install: false
redis:
  cluster:
    enabled: true
registry:
  ingress:
    tls:
      secretName: [redacted]
  storage:
    secret: [redacted]
    key: config

Current behavior

When running the restore, I receive the following output about ten minutes in:

andrew@eikthyrnir:~$ time kubectl exec gitlab-ce-task-runner-6bc6cb786f-dd7cl -n gitlab -it -- backup-utility --restore -t 1608033647_2020_12_15_13.6.3-ee
Unpacking backup
rake aborted!
ActiveRecord::StatementInvalid: PG::DependentObjectsStillExist: ERROR:  cannot drop view pg_stat_statements because extension pg_stat_statements requires it
HINT:  You can drop extension pg_stat_statements instead.
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:92:in `exec'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:92:in `block (2 levels) in execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:91:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract_adapter.rb:722:in `block (2 levels) in log'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract_adapter.rb:721:in `block in log'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract_adapter.rb:712:in `log'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:90:in `execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/marginalia-1.9.0/lib/marginalia.rb:71:in `execute_with_marginalia'
/srv/gitlab/lib/tasks/gitlab/db.rake:40:in `block (4 levels) in <main>'
/srv/gitlab/lib/tasks/gitlab/db.rake:39:in `each'
/srv/gitlab/lib/tasks/gitlab/db.rake:39:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:3:in `load'
/srv/gitlab/bin/bundle:3:in `<main>'

Caused by:
PG::DependentObjectsStillExist: ERROR:  cannot drop view pg_stat_statements because extension pg_stat_statements requires it
HINT:  You can drop extension pg_stat_statements instead.
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:92:in `exec'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:92:in `block (2 levels) in execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:91:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract_adapter.rb:722:in `block (2 levels) in log'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract_adapter.rb:721:in `block in log'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract_adapter.rb:712:in `log'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:90:in `execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/marginalia-1.9.0/lib/marginalia.rb:71:in `execute_with_marginalia'
/srv/gitlab/lib/tasks/gitlab/db.rake:40:in `block (4 levels) in <main>'
/srv/gitlab/lib/tasks/gitlab/db.rake:39:in `each'
/srv/gitlab/lib/tasks/gitlab/db.rake:39:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:3:in `load'
/srv/gitlab/bin/bundle:3:in `<main>'
Tasks: TOP => gitlab:db:drop_tables
(See full trace by running task with --trace)
command terminated with exit code 1

real    11m17.360s
user    0m0.294s
sys     0m0.198s

Expected behavior

The restore to complete without errors.

Versions

  • Chart: 4.6.3
  • Platform:
    • Cloud: Azure VMs, with Kubernetes deployed on them by Rancher.
    • Self-hosted: Rancher RKE
  • Kubernetes: (kubectl version)
    • Client: 1.19.4
    • Server: 1.19.4
  • Helm: (helm version)
    • Client: 3.2
    • Server: 3.2