Geo: Warning: Could not write to the database main: cannot execute UPSERT in a read-only transaction

Problem

In a production environment:

Also, postgresql/current logs contain ERROR: cannot execute INSERT in a read-only transaction.

Also this shows up in a secondary GDK site:

--------------------------------------------------------------------------------
Processing gitlab Geo DB migrations
--------------------------------------------------------------------------------
WARNING: Could not write to the database ci: cannot execute UPSERT in a read-only transaction
WARNING: Could not write to the database main: cannot execute UPSERT in a read-only transaction
geo: == [advisory_lock_connection] object_id: 117980, pg_backend_pid: 43173
geo: == [advisory_lock_connection] object_id: 117980, pg_backend_pid: 43173

This seems to be benign. It can be a red herring.

Proposal

• Avoid the warning
• Avoid the PostgreSQL error altogether

Possible fixes

• Before the offending SQL query, try to detect if the target database is read-only or if the current site is a secondary Geo site.
• If it is, then return early

Implementation Guide

Following the discussions in this thread, a proposed implementation would be to skip the database validation when running migrations and schema load/dumps in the geo namespace.

This amounts to modifying the validate_config.rake task to skip enhancing the migration, dump and load tasks with validation:

ActiveRecord::Tasks::DatabaseTasks.for_each(databases) do |name|
  next if name == 'geo'

  Rake::Task["db:migrate:#{name}"].enhance(['gitlab:db:validate_config'])
  Rake::Task["db:schema:load:#{name}"].enhance(['gitlab:db:validate_config'])
  Rake::Task["db:schema:dump:#{name}"].enhance(['gitlab:db:validate_config'])
end

Writing some tests if possible would be nice.