Can not reset runner tokens

Hello, I migrated gitlab, lost secrets and am trying to access gitlab runners but I get 500 error. I tried all the methods described in https://docs.gitlab.com/ee/raketasks/backup_restore.html#when-the-secrets-file-is-lost, but fixes did not halp. Instead I get errors OpenSSL::Cipher::CipherError (). When running commend:

gitlab-rake gitlab:doctor:secrets VERBOSE=1

I get 2 problems:

I, [2022-04-11T06:53:42.536869 #583396]  INFO -- : - ApplicationSetting failures: 1
D, [2022-04-11T06:53:42.536968 #583396] DEBUG -- : - ApplicationSetting[1]: customers_dot_jwt_signing_key, runners_registration_token
...
I, [2022-04-11T06:53:44.437600 #583396]  INFO -- : - Operations::FeatureFlagsClient failures: 4
D, [2022-04-11T06:53:44.437715 #583396] DEBUG -- : - Operations::FeatureFlagsClient[1]: token
D, [2022-04-11T06:53:44.437772 #583396] DEBUG -- : - Operations::FeatureFlagsClient[2]: token
D, [2022-04-11T06:53:44.437824 #583396] DEBUG -- : - Operations::FeatureFlagsClient[3]: token
D, [2022-04-11T06:53:44.437869 #583396] DEBUG -- : - Operations::FeatureFlagsClient[4]: token

Gitlab info:

Ruby:         ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
GitLab:       14.9.2-ee (3034418fb31) EE
GitLab Shell: 13.24.0
PostgreSQL:   12.7

Tried to set customers_dot_jwt_signing_key to nill, but there is no such field.

Tried to reset runner registration tokes with command:

gitlab-rails runner -e production "ApplicationSetting.current.reset_runners_registration_token!"

But got this result:

Traceback (most recent call last):
61: from bin/rails:4:in `<main>'
60: from bin/rails:4:in `require'
59: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.6/lib/rails/commands.rb:18:in `<top (required)>'
58: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.6/lib/rails/command.rb:48:in `invoke'
57: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.6/lib/rails/command/base.rb:69:in `perform'
56: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
55: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
54: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
53: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.6/lib/rails/commands/runner/runner_command.rb:45:in `perform'
52: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.6/lib/rails/commands/runner/runner_command.rb:45:in `eval'
51: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.1.4.6/lib/rails/commands/runner/runner_command.rb:45:in `<main>'
50: from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable.rb:59:in `block in add_authentication_token_field'
49: from /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:53:in `reset_token'
48: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/suppressor.rb:48:in `save'
47: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/transactions.rb:302:in `save!'
46: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
45: from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
44: from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
43: from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
42: from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
41: from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
40: from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
39: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
38: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
37: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
36: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
35: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
34: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
33: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
32: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
31: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
30: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
29: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/transactions.rb:302:in `block in save!'
28: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/validations.rb:53:in `save!'
27: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/validations.rb:84:in `perform_validations'
26: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/validations.rb:68:in `valid?'
25: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.4.6/lib/active_model/validations.rb:337:in `valid?'
24: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.4.6/lib/active_model/validations/callbacks.rb:117:in `run_validations!'
23: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:824:in `_run_validation_callbacks'
22: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:106:in `run_callbacks'
21: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.4.6/lib/active_model/validations/callbacks.rb:117:in `block in run_validations!'
20: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.4.6/lib/active_model/validations.rb:406:in `run_validations!'
19: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:824:in `_run_validate_callbacks'
18: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:105:in `run_callbacks'
17: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:512:in `invoke_before'
16: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:512:in `each'
15: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:512:in `block in invoke_before'
14: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:199:in `block in halting'
13: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:603:in `block in default_terminator'
12: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:603:in `catch'
11: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
10: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
9: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/callbacks.rb:427:in 'block in make_lambda'
8: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.4.6/lib/active_model/validator.rb:149:in `validate'
7: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.4.6/lib/active_model/validator.rb:149:in `each'
6: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.4.6/lib/active_model/validator.rb:150:in `block in validate'
5: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted.rb:161:in `block (2 levels) in attr_encrypted'
4: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted.rb:329:in `decrypt'
3: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted.rb:240:in `decrypt'
2: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'
1: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final': OpenSSL::Cipher::CipherError

Edited Apr 12, 2022 by Deimantas Aukštkalnis