Error 500 when enabling direct transfer
Summary
Gitlab CE Omnibus 15.9.1-ce.0 on Ubuntu 20.04
gitlab-ctl status:
down: alertmanager: 0s, normally up, want up; run: log: (pid 1888) 429240s
run: gitaly: (pid 29029) 427076s; run: log: (pid 1891) 429240s
run: gitlab-exporter: (pid 29050) 427075s; run: log: (pid 1876) 429240s
run: gitlab-kas: (pid 29055) 427075s; run: log: (pid 1885) 429240s
run: gitlab-workhorse: (pid 29080) 427074s; run: log: (pid 1873) 429240s
run: logrotate: (pid 2166286) 2268s; run: log: (pid 1895) 429240s
run: mailroom: (pid 2174947) 1631s; run: log: (pid 1014035) 84771s
run: node-exporter: (pid 29111) 427074s; run: log: (pid 1875) 429240s
run: postgres-exporter: (pid 29130) 427073s; run: log: (pid 1884) 429240s
run: postgresql: (pid 29142) 427073s; run: log: (pid 1874) 429240s
run: puma: (pid 2175112) 1620s; run: log: (pid 1878) 429240s
run: redis: (pid 29178) 427072s; run: log: (pid 1883) 429240s
run: redis-exporter: (pid 29199) 427071s; run: log: (pid 1877) 429240s
run: registry: (pid 2175302) 1608s; run: log: (pid 2174826) 1638s
run: sidekiq: (pid 2175004) 1627s; run: log: (pid 1879) 429240s
When I try to activate "direct transfer" in admin/application_settings/general#js-visibility-settings, I get a 500 Error with a Request ID. Searching the logs for this Request ID shows the entry in gitlab-rails/production-json.log (see later in this ticket).
Steps to reproduce
- go to Admin Area
- Settings -> General
- Visibility and access controls
- set "Allow migrating GitLab groups and projects by direct transfer" to Enabled
What is the current bug behavior?
Server gives an error 500 without any hints, the log files are not helpful to detect a configuration error or a missing dependency.
The only detectable hint in the log file for me is "OpenSSL::Cipher::CipherError".
What is the expected correct behavior?
At least in the logfile a hint to a missing dependency or a configuration error should be present.
Relevant logs and/or screenshots
gitlab-rails/production_json.log:
{"method":"PATCH",
"path":"/admin/application_settings/general",
"format":"html",
"controller":"Admin::ApplicationSettingsController",
"action":"general",
"status":500,
"time":"2023-03-01T15:49:07.762Z",
"params":[{"key":"_method",
"value":"patch"},
{"key":"authenticity_token",
"value":"[FILTERED]"},
{"key":"application_setting",
"value":{"default_project_creation":"2",
"default_project_visibility":"0",
"default_snippet_visibility":"0",
"default_group_visibility":"0",
"restricted_visibility_levels":[""],
"import_sources":["","github","bitbucket","bitbucket_server","gitlab","fogbugz","git","gitlab_project","gitea","manifest","phabricator"],
"project_export_enabled":"0",
"bulk_import_enabled":"1",
"enabled_git_access_protocol":"",
"custom_http_clone_url_root":"",
"rsa_key_restriction":"0",
"dsa_key_restriction":"0",
"ecdsa_key_restriction":"0",
"ed25519_key_restriction":"0",
"ecdsa_sk_key_restriction":"0",
"ed25519_sk_key_restriction":"0",
"disable_feed_token":"[FILTERED]"}}],
"correlation_id":"faa3073b-5419-4b16-b804-6997703dd4ac",
"meta.caller_id":"Admin::ApplicationSettingsController#general",
"meta.remote_ip":"12.345.678.901",
"meta.feature_category":"not_owned",
"meta.user":"root",
"meta.user_id":1,
"meta.client_id":"user/1",
"remote_ip":"10.239.242.109",
"user_id":1,
"username":"root",
"ua":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36",
"request_urgency":"default",
"target_duration_s":1,
"redis_calls":15,"redis_allowed_cross_slot_calls":1,"redis_duration_s":0.002205,
"redis_read_bytes":2127,"redis_write_bytes":2415,"redis_cache_calls":10,
"redis_cache_duration_s":0.001023,"redis_cache_read_bytes":1944,"redis_cache_write_bytes":622,
"redis_sessions_calls":4,"redis_sessions_allowed_cross_slot_calls":1,"redis_sessions_duration_s":0.000882,
"redis_sessions_read_bytes":183,"redis_sessions_write_bytes":1687,"redis_shared_state_calls":1,
"redis_shared_state_duration_s":0.0003,"redis_shared_state_write_bytes":106,
"db_count":6,"db_write_count":2,
"db_cached_count":0,"db_replica_count":0,
"db_primary_count":6,"db_main_count":6,
"db_main_replica_count":0,"db_replica_cached_count":0,
"db_primary_cached_count":0,"db_main_cached_count":0,
"db_main_replica_cached_count":0,"db_replica_wal_count":0,
"db_primary_wal_count":0,"db_main_wal_count":0,
"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,
"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,
"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,
"db_primary_duration_s":0.005,"db_main_duration_s":0.005,
"db_main_replica_duration_s":0.0,
"cpu_s":0.113725,
"mem_objects":44444,"mem_bytes":5109099,
"mem_mallocs":24058,"mem_total_bytes":6886859,
"pid":2175581,
"worker_id":"puma_3",
"rate_limiting_gates":[],
"exception.class":"OpenSSL::Cipher::CipherError",
"exception.message":"",
"exception.backtrace":["lib/gitlab/database/load_balancing/connection_proxy.rb:120:in 'public_send'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:120:in 'block in write_using_load_balancer'",
"lib/gitlab/database/load_balancing/load_balancer.rb:127:in 'block in read_write'",
"lib/gitlab/database/load_balancing/load_balancer.rb:198:in 'retry_with_backoff'",
"lib/gitlab/database/load_balancing/load_balancer.rb:116:in 'read_write'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:119:in 'write_using_load_balancer'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:71:in 'transaction'",
"app/services/application_settings/update_service.rb:50:in 'update_settings'",
"app/services/application_settings/update_service.rb:12:in 'execute'",
"app/controllers/admin/application_settings_controller.rb:289:in 'perform_update'",
"app/controllers/admin/application_settings_controller.rb:55:in 'block (2 levels) in <class:ApplicationSettingsController>'",
"app/controllers/application_controller.rb:532:in 'set_current_admin'",
"lib/gitlab/session.rb:11:in 'with_session'",
"app/controllers/application_controller.rb:523:in 'set_session_storage'",
"lib/gitlab/i18n.rb:107:in 'with_locale'",
"lib/gitlab/i18n.rb:113:in 'with_user_locale'",
"app/controllers/application_controller.rb:514:in 'set_locale'",
"app/controllers/application_controller.rb:507:in 'set_current_context'",
"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in 'call'",
"lib/gitlab/middleware/memory_report.rb:13:in 'call'",
"lib/gitlab/middleware/speedscope.rb:13:in 'call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in 'call'",
"lib/gitlab/middleware/rails_queue_duration.rb:33:in 'call'",
"lib/gitlab/metrics/rack_middleware.rb:16:in 'block in call'",
"lib/gitlab/metrics/web_transaction.rb:46:in 'run'",
"lib/gitlab/metrics/rack_middleware.rb:16:in 'call'",
"lib/gitlab/jira/middleware.rb:19:in 'call'",
"lib/gitlab/middleware/go.rb:20:in 'call'",
"lib/gitlab/etag_caching/middleware.rb:21:in 'call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in 'block in call'",
"lib/gitlab/database/query_analyzer.rb:37:in 'within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in 'call'",
"lib/gitlab/middleware/multipart.rb:173:in 'call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in 'call'",
"lib/gitlab/middleware/read_only.rb:18:in 'call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in 'call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in 'call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in 'call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in 'call'",
"lib/gitlab/middleware/request_context.rb:21:in 'call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in 'call'",
"config/initializers/fix_local_cache_middleware.rb:11:in 'call'",
"lib/gitlab/middleware/compressed_json.rb:37:in 'call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in 'call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in 'call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:79:in 'call'",
"lib/gitlab/middleware/release_env.rb:13:in 'call'"],
"db_duration_s":0.19403,
"view_duration_s":0.0,
"duration_s":0.29241}
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 20.04 Current User: git Using RVM: no Ruby Version: 2.7.7p221 Gem Version: 3.1.6 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.8 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 15.9.1 Revision: de8f6619031 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.8 URL: https://v000800.xxxx.ch HTTP Clone URL: https://v000800.xxxx.ch/some-group/some-project.git SSH Clone URL: git@v000800.xxxx.ch:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: shibboleth (workaround from #393065 but this issue existed before) GitLab Shell Version: 14.17.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 14.17.0 ? ... OK (14.17.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Checking Reply by email ... IMAP server credentials are correct? ... Checking incoming yes Mailroom enabled? ... skipped MailRoom running? ... skipped Checking Reply by email ... Finished Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 2/1 ... yes 2/2 ... yes 2/3 ... yes 2/4 ... yes 19/5 ... yes 23/6 ... yes 23/7 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.7) Git user has default SSH configuration? ... yes Active users: ... 9 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Checking GitLab App ... Finished Checking GitLab subtasks ... Finished