Skip to content

Better fix for PG::ObjectInUse error when dropping databases

This requires a few things to be true:

  1. Database connections are opened during routes. This is true for the GitLab application. Note this is not allowed anymore in Rails 7.
  2. Multiple databases are configured. With a single database, Rails has always accidentally closed the database connection for the main database.

This fix removes prior attempts to drop the connection(s)

Database connections opened in other initializers are still an issue but Rails kindly closes them for us. Unfortunately Rails has an initializer, set_routes_reloader_hook which is run after config.after_initialize. The sequence goes like this:

  1. Various initializers run, including route loading.
  2. config.after_initialize blocks run (called by Gitlab::Application.finisher_hook). One of the config.after_initialize blocks run is to close all database connections.
  3. Gitlab::Application.set_routes_reloader_hook initializer runs. This loads routes again.

So this fix is simple: we simply call the same thing Rails to close all database connections again. As Rails upstream intended 😄

Related issue: #356656 (closed)

Reproduction

  1. Setup multiple databases following #356353 (closed)
  2. Comment out the enhancement using clear_all_connections in lib/tasks/gitlab/db.rake if in master branch`
  3. Run RAILS_ENV=test be rails db:test:prepare
(arm64) tkgl2:gitlab tkuah$ RAILS_ENV=test be rails db:test:prepare
WARNING: sha256_attribute :fingerprint_sha256 is invalid since the table doesn't exist - you may need to run database migrations

rails aborted!
ActiveRecord::StatementInvalid: PG::ObjectInUse: ERROR:  database "gitlabhq_test_ci" is being accessed by other users
DETAIL:  There is 1 other session using the database.
/Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
/Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
/Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
/Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
/Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:88:in `method_missing'

Caused by:
PG::ObjectInUse: ERROR:  database "gitlabhq_test_ci" is being accessed by other users
DETAIL:  There is 1 other session using the database.
/Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
/Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
/Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
/Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
/Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:88:in `method_missing'
Tasks: TOP => db:test:load => db:test:purge
(See full trace by running task with --trace)
Edited by Thong Kuah

Merge request reports