Skip to content

Migration fails when connecting to existing database with OpenSSL::Cipher::CipherError

Summary

I am attempting to migrate from a Gitlab Server to a Gitlab chart approach. So my database already exists

First connection to the database is made without issues as expected

Attempting to run '/scripts/wait-for-deps /scripts/db-migrate' as a main process
Checking database migrations are up-to-date
Performing migrations (this will initialized if needed)

After that the Performing custom instance setup step fails with an OpenSSL::Cipher::CipherError.

Steps to reproduce

(Please provide the steps to reproduce the issue)

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

This is my values.yaml, both cas have a .crt extension and have just one certificate

      global:
        appConfig:
          ldap:
        certificates:
          customCAs:
          - secret: keos-ca
          - secret: pg-bundle
        gitlab:
          name: gitlab
        hosts:
          domain: mydomain.comm
        ingress:
          class: default-ingress-class
          configureCertmanager: false
        psql:
          database: test-cicdcd-gitlab
          host: pool-gitlab.test-datastores
          password:
            key: password
            secret: database-user
          port: 5432
          ssl:
            clientCertificate: pgbouncer_crt
            clientKey: pgbouncer_key
            secret: pgcertificates
            serverCA: ca.crt
          username: test-cicdcd-gitlab
      nginx-ingress:
        enabled: false
      postgresql:
        install: false
    version: 7.8.0

(Paste sanitized configuration here)

My secrets are configured like this:

apiVersion: v1
items:
- apiVersion: v1
  data:
    ca.crt: --encrypted-secret--
  kind: Secret
  metadata:
    name: keos-ca
    namespace: test-gitlab
  type: Opaque
- apiVersion: v1
  data:
    ca2.crt: --encrypted-secret--
  kind: Secret
  metadata:
    name: pg-bundle
    namespace: test-gitlab
  type: Opaque
- apiVersion: v1
  data:
    ca.crt: --encrypted-secret--
    pgbouncer_crt: --encrypted-secret--
    pgbouncer_key: --encrypted-secret--
  kind: Secret
  metadata:
    name: pgcertificates
    namespace: test-gitlab
  type: Opaque
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Current behavior

Migrations returns this error:

Running db:migrate rake task
main: == [advisory_lock_connection] object_id: 118760, pg_backend_pid: 1424705
main: == [advisory_lock_connection] object_id: 118760, pg_backend_pid: 1424705
Running gitlab:clickhouse:migrate:main rake task
The 'main' ClickHouse database is not configured, skipping migrations
Performing custom instance setup
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final': OpenSSL::Cipher::CipherError
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'
	from /srv/gitlab/lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'
	from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:16:in `decrypt_token'
	from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/encrypted.rb:78:in `get_encrypted_token'
	from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/encrypted.rb:113:in `token_set?'
	from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/base.rb:44:in `ensure_token'
	from /srv/gitlab/app/models/concerns/token_authenticatable_strategies/encrypted.rb:38:in `ensure_token'
	from /srv/gitlab/app/models/concerns/token_authenticatable.rb:49:in `block in add_authentication_token_field'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:400:in `block in make_lambda'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:686:in `catch'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:686:in `block in default_terminator'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:200:in `block in halting'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:595:in `block in invoke_before'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:595:in `each'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:595:in `invoke_before'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:106:in `run_callbacks'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/callbacks.rb:455:in `create_or_update'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/timestamp.rb:126:in `create_or_update'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/persistence.rb:615:in `save'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/validations.rb:47:in `save'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/transactions.rb:298:in `block in save'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
	from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
	from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
	from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
	from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
	from /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
	from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
	from /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/transactions.rb:298:in `save'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/suppressor.rb:50:in `save'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8/lib/active_record/persistence.rb:760:in `update_attribute'
	from /scripts/custom-instance-setup:8:in `<main>'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8/lib/rails/commands/runner/runner_command.rb:43:in `load'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8/lib/rails/commands/runner/runner_command.rb:43:in `perform'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/thor-1.3.0/lib/thor/command.rb:28:in `run'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/thor-1.3.0/lib/thor/invocation.rb:127:in `invoke_command'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/thor-1.3.0/lib/thor.rb:527:in `dispatch'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8/lib/rails/command/base.rb:87:in `perform'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8/lib/rails/command.rb:48:in `invoke'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8/lib/rails/commands.rb:18:in `<main>'
	from <internal:/usr/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
	from <internal:/usr/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
	from /srv/gitlab/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.17.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
	from /srv/gitlab/bin/rails:4:in `<main>'
Disabling authorized keys write in the database.

Expected behavior

Everything should go smoothly

Versions

  • Chart: v.7.8.0
  • Platform:
    • Self-hosted: Kubernetes
  • Kubernetes: (kubectl version)
    • Client: 1.24.10
    • Server: 1.24.10
  • Helm: (helm version)
    • Client: 3.12
    • Server:3.12

Relevant logs

(Please provide any relevate log snippets you have collected, using code blocks (```) to format)

Edited by Alvaro Campesino