Cannot restore to helm-installed version.
Summary
When attempting to restore to a new installation of gitlab (chart version 4.6.3) from an Omnibus installation, the process fails with a series of postgres-related errors.
Steps to reproduce
Backup an omnibus installation, copy the backup to the s3 (minio) bucket for gitlab-backups for the helm chart, and attempt a normal restore.
Configuration used
(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))
certmanager:
install: false
gitlab:
task-runner:
backups:
objectStorage:
config:
secret: [redacted]
key: config
webservice:
ingress:
tls:
secretName: release-gitlab-tls
gitlab-runner:
install: false
global:
appConfig:
objectStorage:
config:
enabled: true
direct_upload: false
background_upload: true
artifacts:
enabled: true
bucket: gitlab-artifacts-storage
connection:
secret: [redacted]
key: connection
backups:
bucket: gitlab-backup-storage
tmpBucket: gitlab-tmp-storage
externalDiffs:
enabled: true
bucket: gitlab-externaldiffs-storage
connection:
secret: [redacted]
key: connection
lfs:
enabled: true
bucket: gitlab-lfs-storage
connection:
secret: [redacted]
key: connection
omniauth:
enabled: true
allowSingleSignOn: true
blockAutoCreatedUsers: false
autoLinkSamlUser: true
autoLinkLdapUser: true
providers:
- secret: [redacted]
key: provider
packages:
enabled: true
bucket: gitlab-packages-storage
connection:
secret: [redacted]
key: connection
pseudonymizer:
enabled: true
bucket: gitlab-pseudonymizer-storage
connection:
secret: [redacted]
key: connection
uploads:
enabled: true
bucket: gitlab-uploads-storage
connection:
secret: [redacted]
key: connection
email:
display_name: [redacted]
from: [redacted]
reply_to: [redacted]
hosts:
domain: [redacted]
gitlab:
name: [redacted]
registry:
name: [redacted]
ingress:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
externalDNS: true
external-dns.alpha.kubernetes.io/ttl: 300
kubernetes.io/tls-acme: true
class: nginx
configureCertmanager: false
minio:
enabled: false
psql:
database: gitlab
host: [redacted]
password:
secret: [redacted]
key: postgresql-password
username: [redacted]
registry:
bucket: gitlab-registry-storage
smtp:
address: [redacted]
authentication: ""
enabled: true
port: 25
tls: none
minio:
ingress:
tls:
secretName: [redacted]
nginx-ingress:
enabled: false
postgresql:
install: false
redis:
cluster:
enabled: true
registry:
ingress:
tls:
secretName: [redacted]
storage:
secret: [redacted]
key: config
Current behavior
When running the restore, I receive the following output about ten minutes in:
andrew@eikthyrnir:~$ time kubectl exec gitlab-ce-task-runner-6bc6cb786f-dd7cl -n gitlab -it -- backup-utility --restore -t 1608033647_2020_12_15_13.6.3-ee
Unpacking backup
rake aborted!
ActiveRecord::StatementInvalid: PG::DependentObjectsStillExist: ERROR: cannot drop view pg_stat_statements because extension pg_stat_statements requires it
HINT: You can drop extension pg_stat_statements instead.
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:92:in `exec'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:92:in `block (2 levels) in execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:91:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract_adapter.rb:722:in `block (2 levels) in log'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract_adapter.rb:721:in `block in log'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract_adapter.rb:712:in `log'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:90:in `execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/marginalia-1.9.0/lib/marginalia.rb:71:in `execute_with_marginalia'
/srv/gitlab/lib/tasks/gitlab/db.rake:40:in `block (4 levels) in <main>'
/srv/gitlab/lib/tasks/gitlab/db.rake:39:in `each'
/srv/gitlab/lib/tasks/gitlab/db.rake:39:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:3:in `load'
/srv/gitlab/bin/bundle:3:in `<main>'
Caused by:
PG::DependentObjectsStillExist: ERROR: cannot drop view pg_stat_statements because extension pg_stat_statements requires it
HINT: You can drop extension pg_stat_statements instead.
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:92:in `exec'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:92:in `block (2 levels) in execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:91:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract_adapter.rb:722:in `block (2 levels) in log'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract_adapter.rb:721:in `block in log'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract_adapter.rb:712:in `log'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:90:in `execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/marginalia-1.9.0/lib/marginalia.rb:71:in `execute_with_marginalia'
/srv/gitlab/lib/tasks/gitlab/db.rake:40:in `block (4 levels) in <main>'
/srv/gitlab/lib/tasks/gitlab/db.rake:39:in `each'
/srv/gitlab/lib/tasks/gitlab/db.rake:39:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:3:in `load'
/srv/gitlab/bin/bundle:3:in `<main>'
Tasks: TOP => gitlab:db:drop_tables
(See full trace by running task with --trace)
command terminated with exit code 1
real 11m17.360s
user 0m0.294s
sys 0m0.198s
Expected behavior
The restore to complete without errors.
Versions
- Chart: 4.6.3
- Platform:
- Cloud: Azure VMs, with Kubernetes deployed on them by Rancher.
- Self-hosted: Rancher RKE
- Kubernetes: (
kubectl version
)- Client: 1.19.4
- Server: 1.19.4
- Helm: (
helm version
)- Client: 3.2
- Server: 3.2