Executing a backup with PgBouncer can cause a full site outage

ZD: https://gitlab.zendesk.com/agent/tickets/99873

Customer has an HA setup with PgBouncer. Upon running the backup Rake task, customer saw these types of errors:

ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR:  relation "keys" does not exist
LINE 1: SELECT  "keys".* FROM "keys" WHERE "keys"."fingerprint" = 'b...

It turns out PostgreSQL 9.6.8 changed the behavior of pg_dump to set a null search path and explicitly include the schema in every SQL query to address CVE-2018-1058 (Uncontrolled search path element in pg_dump and other client applications). When Rails uses the connection, the SQL query fails because the public schema isn't in the search path.

The workaround is to bypass PgBouncer and conduct a backup directly.

Discourse worked around this by adding configuration options to allow the backup task to specify a separate database connection for backups. We should consider doing this too.