Admin page in Geo secondary raises DB loadbalancer exception
As noticed in Sentry by @vsizov and internal-only discussion in Slack, the admin panel (simply secondary.geo/admin/
) will raise a 500.
This is similar to #341847 (closed) and caused by turning on the load balancer by default in !68042 (merged) as @ibaum mentioned in Slack - any transaction will attempt to stick to the primary, writeable database, which doesn't exist on a Geo secondary, resulting in the Failed to determine the write location of the primary database
exception being raised.
Hooking into Gitlab::Database::LoadBalancing::Session#write!
as in #341847 (closed), we can see:
diff --git a/lib/gitlab/database/load_balancing/session.rb b/lib/gitlab/database/load_balancing/session.rb
index 3682c9265c2..17ae92ee06e 100644
--- a/lib/gitlab/database/load_balancing/session.rb
+++ b/lib/gitlab/database/load_balancing/session.rb
@@ -102,6 +102,7 @@ def fallback_to_replicas_for_ambiguous_queries?
end
def write!
+ byebug
@performed_write = true
return if @ignore_writes
Two separate calls and backtraces:
(byebug) bt
--> #0 Gitlab::Database::LoadBalancing::Session.write! at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/load_balancing/session.rb:108
#1 Gitlab::Database::LoadBalancing::ConnectionProxy.transaction(*args#Array, kwargs#Hash, &block#Proc) at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:69
#2 ActiveRecord::Transactions::ClassMethods.transaction(options#Hash, &block#Proc) at /home/catalin/.asdf/installs/ruby/2.7.4/lib/ruby/gems/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:209
#3 block in Gitlab::Database::ActiveRecordBaseTransactionMetrics::ClassMethods.block in transaction(options#Hash, &block#Proc) at /home/catalin/gdk-geo/gitlab/lib/gitlab/database.rb:266
#4 block in #<Class:ActiveSupport::Notifications>.block in instrument(name#String, payload#Hash) at /home/catalin/.asdf/installs/ruby/2.7.4/lib/ruby/gems/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/notifications.rb:203
#5 ActiveSupport::Notifications::Instrumenter.instrument(name#String, payload#Hash) at /home/catalin/.asdf/installs/ruby/2.7.4/lib/ruby/gems/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/notifications/instrumenter.rb:24
#6 #<Class:ActiveSupport::Notifications>.instrument(name#String, payload#Hash) at /home/catalin/.asdf/installs/ruby/2.7.4/lib/ruby/gems/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/notifications.rb:203
#7 Gitlab::Database::ActiveRecordBaseTransactionMetrics::ClassMethods.transaction(options#Hash, &block#Proc) at /home/catalin/gdk-geo/gitlab/lib/gitlab/database.rb:265
#8 Gitlab::Database::Count::ReltuplesCountStrategy.size_estimates(check_statistics#FalseClass) at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/count/reltuples_count_strategy.rb:57
#9 Gitlab::Database::Count::TablesampleCountStrategy.count at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/count/tablesample_count_strategy.rb:21
#10 block in #<Class:Gitlab::Database::Count>.block in approximate_counts(models#Array, strategies#Array) at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/count.rb:44
ͱ-- #11 Array.each at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/count.rb:39
ͱ-- #12 Enumerable.each_with_object() at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/count.rb:39
#13 #<Class:Gitlab::Database::Count>.approximate_counts(models#Array, strategies#Array) at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/count.rb:39
#14 Admin::DashboardController.index at /home/catalin/gdk-geo/gitlab/app/controllers/admin/dashboard_controller.rb:12
#15 EE::Admin::DashboardController.index at /home/catalin/gdk-geo/gitlab/ee/app/controllers/ee/admin/dashboard_controller.rb:12
(byebug) bt
--> #0 Gitlab::Database::LoadBalancing::Session.write! at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/load_balancing/session.rb:108
#1 Gitlab::Database::LoadBalancing::ConnectionProxy.transaction(*args#Array, kwargs#Hash, &block#Proc) at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:69
#2 ActiveRecord::Transactions::ClassMethods.transaction(options#Hash, &block#Proc) at /home/catalin/.asdf/installs/ruby/2.7.4/lib/ruby/gems/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:209
#3 block in Gitlab::Database::ActiveRecordBaseTransactionMetrics::ClassMethods.block in transaction(options#Hash, &block#Proc) at /home/catalin/gdk-geo/gitlab/lib/gitlab/database.rb:266
#4 block in #<Class:ActiveSupport::Notifications>.block in instrument(name#String, payload#Hash) at /home/catalin/.asdf/installs/ruby/2.7.4/lib/ruby/gems/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/notifications.rb:203
#5 ActiveSupport::Notifications::Instrumenter.instrument(name#String, payload#Hash) at /home/catalin/.asdf/installs/ruby/2.7.4/lib/ruby/gems/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/notifications/instrumenter.rb:24
#6 #<Class:ActiveSupport::Notifications>.instrument(name#String, payload#Hash) at /home/catalin/.asdf/installs/ruby/2.7.4/lib/ruby/gems/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/notifications.rb:203
#7 Gitlab::Database::ActiveRecordBaseTransactionMetrics::ClassMethods.transaction(options#Hash, &block#Proc) at /home/catalin/gdk-geo/gitlab/lib/gitlab/database.rb:265
#8 Gitlab::Database::Count::ReltuplesCountStrategy.size_estimates(check_statistics#TrueClass) at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/count/reltuples_count_strategy.rb:57
#9 Gitlab::Database::Count::ReltuplesCountStrategy.count at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/count/reltuples_count_strategy.rb:25
#10 block in #<Class:Gitlab::Database::Count>.block in approximate_counts(models#Array, strategies#Array) at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/count.rb:44
ͱ-- #11 Array.each at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/count.rb:39
ͱ-- #12 Enumerable.each_with_object() at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/count.rb:39
#13 #<Class:Gitlab::Database::Count>.approximate_counts(models#Array, strategies#Array) at /home/catalin/gdk-geo/gitlab/lib/gitlab/database/count.rb:39
#14 Admin::DashboardController.index at /home/catalin/gdk-geo/gitlab/app/controllers/admin/dashboard_controller.rb:12
#15 EE::Admin::DashboardController.index at /home/catalin/gdk-geo/gitlab/ee/app/controllers/ee/admin/dashboard_controller.rb:12
We can see that Gitlab::Database::Count#approximate_counts
is the culprit, the ReltuplesCountStrategy opens a transaction so this attempts to stick to the primary.
We should handle this differently when the DB is read-only, maybe excluding this strategy from the #approximate_counts
call.