Internal server error 500 after upgrading FOSS release 13.5.4 to 13.6 stable
Summary
Opening the sign-in page or trying to clone a project results in an internal server error 500.
Sidekiq process dies shortly after being started.
Steps to reproduce
Example Project
What is the current bug behavior?
What is the expected correct behavior?
Relevant logs and/or screenshots
production.log right after starting gitlab (from /etc/init.d/gitlab):
Raven 3.0.4 configured not to capture errors: DSN not set
Creating scope :without_statuses. Overwriting existing method CommitStatus.without_statuses.
An enum element in Ci::Runner uses the prefix 'not_'. This will cause a conflict with auto generated negative scopes.
Creating scope :with_api_entity_associations. Overwriting existing method Issue.with_api_entity_associations.
Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects.
Creating scope :join_project. Overwriting existing method MergeRequest.join_project.
Creating scope :references_project. Overwriting existing method MergeRequest.references_project.
Creating scope :system. Overwriting existing method Note.system.
Creating scope :order_created_desc. Overwriting existing method Packages::Package.order_created_desc.
Creating scope :order_name_desc. Overwriting existing method Packages::Package.order_name_desc.
Creating scope :without_statuses. Overwriting existing method Ci::Pipeline.without_statuses.
An enum element in UserStatus uses the prefix 'not_'. This will cause a conflict with auto generated negative scopes.
Creating scope :open. Overwriting existing method AlertManagement::Alert.open.
Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc.
Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses.
Creating scope :created. Overwriting existing method Ci::Stage.created.
Creating scope :pending. Overwriting existing method Ci::Stage.pending.
Creating scope :running. Overwriting existing method Ci::Stage.running.
Creating scope :success. Overwriting existing method Ci::Stage.success.
Creating scope :failed. Overwriting existing method Ci::Stage.failed.
Creating scope :canceled. Overwriting existing method Ci::Stage.canceled.
Creating scope :skipped. Overwriting existing method Ci::Stage.skipped.
Creating scope :manual. Overwriting existing method Ci::Stage.manual.
Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled.
Creating scope :preparing. Overwriting existing method Ci::Stage.preparing.
Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource.
Creating scope :open. Overwriting existing method ExternalPullRequest.open.
Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.
Creating scope :with_user. Overwriting existing method GroupMember.with_user.
Creating scope :system. Overwriting existing method PrometheusMetric.system.
Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system.
Started GET "/" for at 2020-11-24 12:35:09 +0100
Processing by RootController#index as HTML
Completed 500 Internal Server Error in 3ms (Elasticsearch: 0.0ms | Allocations: 1130)
ArgumentError (wrong number of arguments (given 2, expected 1)):
lib/gitlab/current_settings.rb:20:in `method_missing'
lib/gitlab/auth/two_factor_auth_verifier.rb:13:in `two_factor_authentication_required?'
app/controllers/concerns/enforces_two_factor_authentication.rb:32:in `two_factor_authentication_required?'
app/controllers/concerns/enforces_two_factor_authentication.rb:26:in `check_two_factor_requirement'
app/controllers/application_controller.rb:477:in `block in set_current_context'
lib/gitlab/application_context.rb:54:in `block in use'
lib/gitlab/application_context.rb:54:in `use'
lib/gitlab/application_context.rb:21:in `with_context'
app/controllers/application_controller.rb:469:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:234:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
config.ru:32:in `block (2 levels) in <main>'
Raven 3.0.4 configured not to capture errors: DSN not setsidekiq.log:
Process doesn't exist
WARNING: PID file creation will be removed in Sidekiq 6.0, see #4045. Please use a proper process supervisor to start and manage your services
WARNING: Daemonization mode will be removed in Sidekiq 6.0, see #4045. Please use a proper process supervisor to start and manage your services
WARNING: Logfile redirection will be removed in Sidekiq 6.0, see #4045. Sidekiq will only log to STDOUT
{"severity":"INFO","time":"2020-11-24T12:12:13.361Z","message":"Booting Sidekiq 5.2.9 with redis options {:instrumentation_class=\u003eGitlab::Instrumentation::Redis::Queues, :path=\u003e\"/var/run/redis/gitlab.sock\", :namespace=\u003e\"resque:gitlab\", :id=\u003e\"Sidekiq-server-PID-18948\", :url=\u003enil}"}
{"severity":"INFO","time":"2020-11-24T12:12:13.371Z","message":"GitLab reliable fetch activated!"}
wrong number of arguments (given 1, expected 0)
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.3/lib/active_record/attribute_methods/read.rb:15:in `uuid'
/home/gitlab/gitlab/lib/gitlab/current_settings.rb:20:in `method_missing'
/home/gitlab/gitlab/config/settings.rb:192:in `cron_for_usage_ping'
/home/gitlab/gitlab/config/settings.rb:156:in `load_dynamic_cron_schedules!'
/home/gitlab/gitlab/config/initializers/sidekiq.rb:66:in `block in <top (required)>'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq.rb:75:in `configure_server'
/home/gitlab/gitlab/config/initializers/sidekiq.rb:29:in `<top (required)>'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:318:in `load'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:318:in `block in load'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:291:in `load_dependency'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:318:in `load'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.3/lib/rails/engine.rb:666:in `block in load_config_initializer'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.3/lib/active_support/notifications.rb:182:in `instrument'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.3/lib/rails/engine.rb:665:in `load_config_initializer'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.3/lib/rails/engine.rb:625:in `block (2 levels) in <class:Engine>'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.3/lib/rails/engine.rb:624:in `each'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.3/lib/rails/engine.rb:624:in `block in <class:Engine>'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.3/lib/rails/initializable.rb:32:in `instance_exec'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.3/lib/rails/initializable.rb:32:in `run'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.3/lib/rails/initializable.rb:61:in `block in run_initializers'
/opt/ruby-2.6.6/lib64/ruby/2.6.0/tsort.rb:228:in `block in tsort_each'
/opt/ruby-2.6.6/lib64/ruby/2.6.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/ruby-2.6.6/lib64/ruby/2.6.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/opt/ruby-2.6.6/lib64/ruby/2.6.0/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/ruby-2.6.6/lib64/ruby/2.6.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.3/lib/rails/initializable.rb:50:in `each'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.3/lib/rails/initializable.rb:50:in `tsort_each_child'
/opt/ruby-2.6.6/lib64/ruby/2.6.0/tsort.rb:415:in `each_strongly_connected_component_from'
/opt/ruby-2.6.6/lib64/ruby/2.6.0/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/ruby-2.6.6/lib64/ruby/2.6.0/tsort.rb:347:in `each'
/opt/ruby-2.6.6/lib64/ruby/2.6.0/tsort.rb:347:in `call'
/opt/ruby-2.6.6/lib64/ruby/2.6.0/tsort.rb:347:in `each_strongly_connected_component'
/opt/ruby-2.6.6/lib64/ruby/2.6.0/tsort.rb:226:in `tsort_each'
/opt/ruby-2.6.6/lib64/ruby/2.6.0/tsort.rb:205:in `tsort_each'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.3/lib/rails/initializable.rb:60:in `run_initializers'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.3/lib/rails/application.rb:363:in `initialize!'
/home/gitlab/gitlab/config/environment.rb:5:in `<top (required)>'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:324:in `require'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:324:in `block in require'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:291:in `load_dependency'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.3/lib/active_support/dependencies.rb:324:in `require'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/cli.rb:288:in `boot_system'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/lib/sidekiq/cli.rb:46:in `run'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.9/bin/sidekiq:12:in `<top (required)>'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/bin/sidekiq:23:in `load'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/bin/sidekiq:23:in `<top (required)>'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/ruby-2.6.6/lib64/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/home/gitlab/bin/bundle:23:in `load'
/home/gitlab/bin/bundle:23:in `<main>'Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
System information
System:
Current User: gitlab
Using RVM: no
Ruby Version: 2.6.6p146
Gem Version: 3.0.3
Bundler Version:2.1.4
Rake Version: 13.0.1
Redis Version: 6.0.4
Git Version: 2.29.2
Sidekiq Version:5.2.9
Go Version: go1.14.12 linux/amd64
rake aborted!
ArgumentError: wrong number of arguments (given 1, expected 0)
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.3/lib/active_record/attribute_methods/read.rb:15:in `default_project_visibility'
/home/gitlab/gitlab/lib/gitlab/current_settings.rb:20:in `method_missing'
/home/gitlab/gitlab/app/models/project.rb:798:in `initialize'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.3/lib/active_record/inheritance.rb:70:in `new'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.3/lib/active_record/inheritance.rb:70:in `new'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.3/lib/active_record/reflection.rb:158:in `build_association'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.3/lib/active_record/associations/association.rb:321:in `build_record'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.3/lib/active_record/associations/collection_association.rb:108:in `build'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.3.3/lib/active_record/associations/collection_proxy.rb:316:in `build'
/home/gitlab/gitlab/lib/tasks/gitlab/info.rake:48:in `block (3 levels) in <top (required)>'
/home/gitlab/gitlab/vendor/bundle/ruby/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/home/gitlab/bin/bundle:23:in `load'
/home/gitlab/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:env:info
(See full trace by running task with --trace)Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)(we will only investigate if the tests are passing)