Devise initializer runs expensive database queries on primary database
After !132023 (merged), the initializer 08_devise.rb
loads application settings before the database load balancer has initialized. This causes a very expensive query that peaks to roughly 4 cpu cores of saturation (queryid 4396144889388454391
) to run on the primary database during application boot, when it could otherwise be pushed to a replica once load balancing is set up:
This is caused by the change in !132023 (merged) , as shown by the following stacktrace:
Stacktrace
column_definitions [postgresql_adapter.rb:916] (ActiveRecord::ConnectionAdapters::PostgreSQLAdapter) << runs the expensive query
columns [schema_statements.rb:117] (ActiveRecord::ConnectionAdapters::SchemaStatements)
columns [schema_cache.rb:117] (ActiveRecord::ConnectionAdapters::SchemaCache)
columns [schema_cache.rb:116] (ActiveRecord::ConnectionAdapters::SchemaCache)
columns_hash [schema_cache.rb:125] (ActiveRecord::ConnectionAdapters::SchemaCache)
columns_hash [schema_cache.rb:124] (ActiveRecord::ConnectionAdapters::SchemaCache)
load_schema! [model_schema.rb:580] (ActiveRecord::ModelSchema::ClassMethods)
load_schema! [attributes.rb:264] (ActiveRecord::Attributes::ClassMethods)
load_schema! [encryptable_record.rb:122] (ActiveRecord::Encryption::EncryptableRecord)
load_schema [model_schema.rb:566] (ActiveRecord::ModelSchema::ClassMethods)
load_schema [model_schema.rb:563] (ActiveRecord::ModelSchema::ClassMethods)
columns_hash [model_schema.rb:419] (ActiveRecord::ModelSchema::ClassMethods)
attribute_instance_methods_as_symbols [active_record.rb:94] (AttrEncrypted::Adapters::ActiveRecord)
attr_encrypted [attr_encrypted.rb:147] (AttrEncrypted) << AttrEncrypted queries for column definitions (expensive)
attr_encrypted [attr_encrypted.rb:144] (AttrEncrypted)
attr_encrypted [active_record.rb:53] (AttrEncrypted::Adapters::ActiveRecord)
<class:ApplicationSetting> [application_setting.rb:687] << The ApplicationSetting model is loaded before the load balancer is set up
<main> [application_setting.rb:3]
require [kernel_require.rb:30] (Kernel)
require [kernel.rb:30] (Kernel)
cached_application_settings [application_setting_fetcher.rb:28] (singleton class of Gitlab::ApplicationSettingFetcher)
current_application_settings [application_setting_fetcher.rb:11] (singleton class of Gitlab::ApplicationSettingFetcher)
current_application_settings [current_settings.rb:15] (singleton class of Gitlab::CurrentSettings)
fetch [null_store.rb:37] (Gitlab::SafeRequestStore::NullStore)
fetch [forwardable.rb:240]
current_application_settings [current_settings.rb:15] (singleton class of Gitlab::CurrentSettings)
method_missing [current_settings.rb:29] (singleton class of Gitlab::CurrentSettings)
block in <main> [8_devise.rb:165] << The devise initializer queries an application setting before the load balancer is set up.
setup [devise.rb:314] (Devise)
<main> [8_devise.rb:7]
load_config_initializer [engine.rb:667] (Rails::Engine)
instrument [notifications.rb:208] (singleton class of ActiveSupport::Notifications)
load_config_initializer [engine.rb:666] (Rails::Engine)
block (2 levels) in <class:Engine> [engine.rb:620]
block in <class:Engine> [engine.rb:619]
run [initializable.rb:32] (Rails::Initializable::Initializer)
run_initializers [initializable.rb:61] (Rails::Initializable)
tsort_each [tsort.rb:228] (TSort)
each_strongly_connected_component [tsort.rb:350] (TSort)
each_strongly_connected_component_from [tsort.rb:422] (TSort)
each_strongly_connected_component_from [tsort.rb:431] (TSort)
each_strongly_connected_component_from [tsort.rb:421] (TSort)
tsort_each_child [initializable.rb:50] (Rails::Initializable::Collection)
each_strongly_connected_component_from [tsort.rb:415] (TSort)
each_strongly_connected_component [tsort.rb:349] (TSort)
each_strongly_connected_component [tsort.rb:347] (TSort)
tsort_each [tsort.rb:226] (TSort)
tsort_each [tsort.rb:205] (TSort)
run_initializers [initializable.rb:60] (Rails::Initializable)
initialize! [application.rb:372] (Rails::Application)
<main> [environment.rb:7]
require [kernel_require.rb:30] (Kernel)
require [kernel.rb:38] (Kernel)
block in <main> [config.ru:5]
new_from_string [builder.rb:116] (Rack::Builder)
load_file [builder.rb:105] (Rack::Builder)
parse_file [builder.rb:66] (Rack::Builder)
build_app_and_options_from_config [server.rb:349] (Rack::Server)
app [server.rb:249] (Rack::Server)
wrapped_app [server.rb:422] (Rack::Server)
log_to_stdout [server_command.rb:76] (Rails::Server)
start [server_command.rb:36] (Rails::Server)
perform [server_command.rb:143] (Rails::Command::ServerCommand)
perform [server_command.rb:134] (Rails::Command::ServerCommand)
run [command.rb:28] (Thor::Command)
invoke_command [invocation.rb:127] (Thor::Invocation)
dispatch [thor.rb:527] (singleton class of Thor)
perform [base.rb:87] (singleton class of Rails::Command::Base)
invoke [command.rb:48] (singleton class of Rails::Command)
<main> [commands.rb:18]
require [kernel_require.rb:30] (Kernel)
<top (required)> [rails:4]
To fix this, the code needs to change to not query a setting until after config/initializers/load_balancing.rb
runs and sets up the load balancer.