Geo: Initialization order issue with Geo database

Summary

On a Geo secondary node, the Sidekiq connection pool is getting set to 1 instead of threads+1. This is leading to could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use" errors in Sidekiq.

Steps to reproduce

Noticed after upgrading a Geo install to 12.7.2.

What is the current bug behavior?

In this line, Gitlab::Geo.geo_database_configured? is being evaluated before the Geo database has been initialized.

What is the expected correct behavior?

The Geo database should be initialized first so that they Sidekiq pool can be set correctly.

Relevant logs and/or screenshots

From /var/log/gitlab/sidekiq/current:

{"severity":"WARN","time":"2020-01-29T22:49:31.440Z","error_class":"ActiveRecord::ConnectionTimeoutError","error_message":"could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use","context":"Job raised exception","jobstr":"{\"queue\":\"cronjob:geo_metrics_update\",\"args\":[],\"class\":\"Geo::MetricsUpdateWorker\",\"retry\":false,\"queue_namespace\":\"cronjob\",\"jid\":\"be2a48565605087ef8a3383c\",\"created_at\":1580338161.3469946,\"correlation_id\":\"50a4455796621a86f54f1e18fb5d8472\",\"enqueued_at\":1580338161.3474455}","queue":"cronjob:geo_metrics_update","args":[],"class":"Geo::MetricsUpdateWorker","retry":false,"queue_namespace":"cronjob","jid":"be2a48565605087ef8a3383c","created_at":"2020-01-29T22:49:21.346Z","correlation_id":"50a4455796621a86f54f1e18fb5d8472","enqueued_at":"2020-01-29T22:49:21.347Z","error_backtrace":["ee/app/models/geo/tracking_base.rb:30:in `connection'","ee/app/models/geo/event_log_state.rb:9:in `last_processed'","ee/app/models/geo_node_status.rb:188:in `current_cursor_last_event_id'","ee/app/models/geo_node_status.rb:300:in `load_secondary_data'","ee/app/models/geo_node_status.rb:179:in `load_data_from_current_node'","ee/app/models/geo_node_status.rb:111:in `current_node_status'","ee/app/services/geo/metrics_update_service.rb:24:in `current_node_status'","ee/app/services/geo/metrics_update_service.rb:10:in `execute'","ee/app/workers/geo/metrics_update_worker.rb:14:in `block in perform'","app/services/concerns/exclusive_lease_guard.rb:29:in `try_obtain_lease'","ee/app/workers/geo/metrics_update_worker.rb:14:in `perform'","lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'"]}

Customer ticket: https://gitlab.zendesk.com/agent/tickets/145499 (internal use)

I was also able to reproduce this by upgrading to 12.7.2

Possible fixes

Move the Geo database init into https://gitlab.com/gitlab-org/gitlab/blob/7a32e84e62e73828d6ea9c8e3052eb08c00ed5bc/config/initializers/database_config.rb

/cc @dbalexandre