Backup Troubleshooting: Handle Deploy Tokens in "When the secrets file is lost"
Summary
Hello,
After a complete crash of my k8s cluster, I deployed a new instance of gitlab and restored the backup. No issue so far. However, I tried to redeploy runners and when I access the CI/CD page of my projects or the Admin > CI/CD > Runners page, I have a 500 error and the following trace in the webserver logs:
"gitlab","subcomponent":"production_json","level":"error","method":"GET","path":"/admin/runners","format":"html","controller":"Admin::RunnersController","action":"index","status":500,"time":"2024-04-26T13:37:05.062Z","params":[],"correlation_id":"01HWD9ZCYEA62GRAVH8KG986AP","meta.caller_id":"Admin::RunnersController#index","meta.remote_ip":"10.42.3.12","meta.feature_category":"runner","meta.user":"clement","meta.user_id":2,"meta.client_id":"user/2","remote_ip":"10.42.3.12","user_id":2,"username":"clement","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36","queue_duration_s":0.008051,"request_urgency":"low","target_duration_s":5,"redis_calls":10,"redis_allowed_cross_slot_calls":1,"redis_duration_s":0.011545,"redis_read_bytes":1428,"redis_write_bytes":998,"redis_cache_calls":6,"redis_cache_duration_s":0.008856,"redis_cache_read_bytes":1217,"redis_cache_write_bytes":355,"redis_sessions_calls":3,"redis_sessions_allowed_cross_slot_calls":1,"redis_sessions_duration_s":0.002146,"redis_sessions_read_bytes":211,"redis_sessions_write_bytes":590,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000543,"redis_shared_state_write_bytes":53,"db_count":7,"db_write_count":0,"db_cached_count":1,"db_replica_count":0,"db_primary_count":7,"db_main_count":7,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":1,"db_main_cached_count":1,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.009,"db_main_duration_s":0.009,"db_main_replica_duration_s":0.0,"cpu_s":0.065774,"mem_objects":23304,"mem_bytes":2523408,"mem_mallocs":7996,"mem_total_bytes":3455568,"pid":819,"worker_id":"puma_1","rate_limiting_gates":[],"exception.class":"ActionView::Template::Error","exception.message":"","exception.backtrace":["lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'","app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:18:in `decrypt_token'","app/models/concerns/token_authenticatable_strategies/encrypted.rb:78:in `get_encrypted_token'","app/models/concerns/token_authenticatable_strategies/encrypted.rb:113:in `token_set?'","app/models/concerns/token_authenticatable_strategies/base.rb:50:in `ensure_token!'","app/models/concerns/token_authenticatable.rb:54:in `block in add_authentication_token_field'","app/models/application_setting_implementation.rb:460:in `runners_registration_token'","lib/gitlab/current_settings.rb:32:in `method_missing'","app/helpers/ci/runners_helper.rb:67:in `admin_runners_data_attributes'","app/views/admin/runners/index.html.haml:4","app/controllers/application_controller.rb:142:in `render'","ee/lib/gitlab/ip_address_state.rb:10:in `with'","ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'","app/controllers/application_controller.rb:524:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:515:in `set_session_storage'","lib/gitlab/i18n.rb:107:in `with_locale'","lib/gitlab/i18n.rb:113:in `with_user_locale'","app/controllers/application_controller.rb:506:in `set_locale'","app/controllers/application_controller.rb:499:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:37:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:37:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'","lib/gitlab/middleware/release_env.rb:13:in `call'"],"exception.cause_class":"OpenSSL::Cipher::CipherError","db_duration_s":0.00696,"view_duration_s":0.0,"duration_s":0.06242}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":" "}
{"component": "gitlab","subcomponent":"production","level":"error","time":"2024-04-26T13:37:05Z","message":"ActionView::Template::Error ():"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":" 1: - breadcrumb_title _('Runners')"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":" 2: - page_title _('Runners')"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":" 3: "}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":" 4: #js-admin-runners{ data: admin_runners_data_attributes }"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":" "}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:18:in `decrypt_token'"}
{"component": "gitlab","subcomponent":"production","level":"info","time":"2024-04-26T13:37:05Z","message":"app/models/concerns/token_authenticatable_strategies/encrypted.rb:78:in `get_encrypted_token'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"app/models/concerns/token_authenticatable_strategies/encrypted.rb:113:in `token_set?'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"app/models/concerns/token_authenticatable_strategies/base.rb:50:in `ensure_token!'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"app/models/concerns/token_authenticatable.rb:54:in `block in add_authentication_token_field'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"app/models/application_setting_implementation.rb:460:in `runners_registration_token'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/current_settings.rb:32:in `method_missing'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"app/helpers/ci/runners_helper.rb:67:in `admin_runners_data_attributes'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"app/views/admin/runners/index.html.haml:4"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"app/controllers/application_controller.rb:142:in `render'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"ee/lib/gitlab/ip_address_state.rb:10:in `with'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"app/controllers/application_controller.rb:524:in `set_current_admin'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/session.rb:11:in `with_session'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"app/controllers/application_controller.rb:515:in `set_session_storage'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/i18n.rb:107:in `with_locale'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/i18n.rb:113:in `with_user_locale'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"app/controllers/application_controller.rb:506:in `set_locale'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"app/controllers/application_controller.rb:499:in `set_current_context'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/memory_report.rb:13:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/speedscope.rb:13:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/metrics/web_transaction.rb:46:in `run'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/metrics/rack_middleware.rb:16:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/jira/middleware.rb:19:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/go.rb:20:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/etag_caching/middleware.rb:21:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/database/query_analyzer.rb:37:in `within'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/query_analyzer.rb:11:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/multipart.rb:173:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/read_only/controller.rb:50:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/read_only.rb:18:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/basic_health_check.rb:25:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"error","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/request_context.rb:21:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"config/initializers/fix_local_cache_middleware.rb:11:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/compressed_json.rb:37:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'"}
{"component": "gitlab","subcomponent":"production","level":"unknown","time":"2024-04-26T13:37:05Z","message":"lib/gitlab/middleware/release_env.rb:13:in `call'"}
Results of GitLab application Check
git@gitlab-toolbox-78b8d5f9c5-tkn2m:/$ gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.18.0 ? ... OK (14.18.0)
Running /home/git/gitlab-shell/bin/check
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /home/git/gitlab-shell/config.yml
Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... no
Try fixing it:
Install the Service
For more information see:
doc/install/installation.md in section "Install the Service"
Please fix the error above and rerun the checks.
Systemd unit files or init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
2/1 ... yes
5/3 ... yes
5/4 ... yes
5/5 ... yes
5/6 ... yes
5/7 ... yes
5/8 ... yes
5/10 ... yes
5/11 ... yes
5/12 ... yes
23/13 ... yes
5/16 ... yes
23/17 ... yes
5/18 ... yes
23/19 ... yes
36/20 ... yes
36/21 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (3.0.6)
Git user has default SSH configuration? ... yes
Active users: ... 6
Is authorized keys file accessible? ... skipped (authorized keys not enabled)
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished