Restore from OmniBus gives error 500 in Admin.

Summary

Restoring a gitlab 11.7.5 backup from an (Kubernetes) OmniBus install into a fresh cloud native helm installation works seemingly fine, but saving settings in Admin -> Settings and accessing Admin -> Overview -> Runners results in error 500.

Steps to reproduce

  1. Backup old OmniBus installation using rake
  2. Copy backup to a local minio, and generate share link
  3. helm install gitlab
  4. kubctl exec task-runner -- backup-utility --restore -f minio_share_link
  5. Log in as root
  6. Admin -> Settings -> ... -> Save changes -> Error 500

During the backup restore I get the following warning:

WARNING: This version of GitLab depends on gitlab-shell 8.4.4, but you're running Unknown. Please update gitlab-shell.

Configuration used

Latest cloud native helm chart with the following modifications to values.yaml:

  • gitlabVersion: 11.7.5
  • LDAP enabled
  • ingress.configureCertmanager=false (using external cert-manager and custer issuer)

Current behavior

Admin -> Settings and Admin -> Overview -> Runners not working properly, error 500.

Expected behavior

Being able to change and save settings under Admin

Versions

  • Chart: 64cac4a6
  • Platform:
    • Self-hosted: NixOS 19.03pre169779.b75eab88bde (Koi)
  • Kubernetes: (kubectl version)
    • Client: 1.13.3
    • Server: 1.13.3
  • Helm: (helm version)
    • Client: 2.11.0
    • Server: 2.12.3

Relevant logs

This is the log output on the unicorn pod when pressing (any) "Save changes":

==> /var/log/gitlab/production.log <==
  
OpenSSL::Cipher::CipherError ():
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:55:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:42:in `ensure_token'
app/models/concerns/token_authenticatable.rb:38:in `block in add_authentication_token_field'
app/services/application_settings/update_service.rb:18:in `execute'
lib/gitlab/metrics/instrumentation.rb:161:in `block in execute'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `execute'
app/controllers/admin/application_settings_controller.rb:40:in `update'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:417:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:24:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:55:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

Trying to access Admin -> Overview -> Runners:

==> /var/log/gitlab/production.log <==
  
ActionView::Template::Error ():
    37: 
    38:     .col-sm-6
    39:       .bs-callout
    40:         = render partial: 'ci/runner/how_to_setup_runner',
    41:                  locals: { registration_token: Gitlab::CurrentSettings.runners_registration_token,
    42:                            type: 'shared',
    43:                            reset_token_url: reset_registration_token_admin_application_settings_path }
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:55:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:33:in `ensure_token!'
app/models/concerns/token_authenticatable.rb:43:in `block in add_authentication_token_field'
app/models/application_setting.rb:410:in `runners_registration_token'
lib/gitlab/current_settings.rb:15:in `method_missing'
app/views/admin/runners/index.html.haml:40:in `_app_views_admin_runners_index_html_haml__2137852246200819084_47281105320380'
app/controllers/application_controller.rb:113:in `render'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:417:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:24:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:55:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Started GET "/admin" for 127.0.0.1 at 2019-02-21 10:30:21 +0000
Processing by Admin::DashboardController#index as HTML
unknown OID 28: failed to recognize type of 'relfrozenxid'. It will be treated as String.
unknown OID 1034: failed to recognize type of 'relacl'. It will be treated as String.
Completed 200 OK in 414ms (Views: 211.0ms | ActiveRecord: 152.0ms)