Geo: After failing over to secondary, Geo Sidekiq jobs failed ungracefully

After we promoted GSTG to the primary, we ran gitlab-ctl reconfigure, which now gets rid of database_geo.yml. Unfortunately, this led to lots of failed Sidekiq jobs with this backtrace:

{
  "severity": "WARN",
  "time": "2018-06-15T15:43:45.536Z",
  "message": "/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/geo/tracking_base.rb:15:in `connection'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/attributes.rb:93:in `columns'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/attributes.rb:98:in `columns_hash'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/query_methods.rb:969:in `block in create_binds'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/query_methods.rb:967:in `each'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/query_methods.rb:967:in `partition'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/query_methods.rb:967:in `create_binds'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/query_methods.rb:954:in `build_where'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/query_methods.rb:584:in `where!'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/query_methods.rb:574:in `where'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation/finder_methods.rb:84:in `find_by'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/relation.rb:224:in `find_or_initialize_by'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:6:in `find_or_initialize_by'\n/opt/gitlab/embedded/service/gitlab-rails/ee/app/services/geo/file_download_service.rb:55:in `update_registry'\n/opt/gitlab/embedded/service/gitlab-rails/ee/app/services/geo/file_download_service.rb:21:in `block in execute'\n/opt/gitlab/embedded/service/gitlab-rails/app/services/concerns/exclusive_lease_guard.rb:22:in `try_obtain_lease'\n/opt/gitlab/embedded/service/gitlab-rails/ee/app/services/geo/file_download_service.rb:13:in `execute'\n/opt/gitlab/embedded/service/gitlab-rails/ee/app/workers/geo/file_download_worker.rb:9:in `perform'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:187:in `execute_job'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:169:in `block (2 levels) in process'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/server_middleware.rb:5:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:6:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/shutdown.rb:52:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_middleware.rb:13:in `block in call'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/transaction.rb:53:in `run'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_middleware.rb:13:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/server/active_record.rb:16:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sentry-raven-2.7.2/lib/raven/integrations/sidekiq.rb:9:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:133:in `invoke'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:168:in `block in process'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:139:in `block (6 levels) in dispatch'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:98:in `local'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:138:in `block (5 levels) in dispatch'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq.rb:36:in `block in <module:Sidekiq>'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:134:in `block (4 levels) in dispatch'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:199:in `stats'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:129:in `block (3 levels) in dispatch'\n/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_logging/structured_logger.rb:13:in `call'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:128:in `block (2 levels) in dispatch'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:73:in `global'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:127:in `block in dispatch'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:48:in `with_context'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:42:in `with_job_hash_context'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:126:in `dispatch'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:167:in `process'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:85:in `process_one'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:73:in `run'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:16:in `watchdog'\n/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:25:in `block in safe_thread'"
}