testdb: Fix stuck PgBouncer connections by killing via admin command
The PgBouncer-based test job is really flaky, where the symptom of the flakiness is that we fail to close all connections to the database on test cleanup. Seemingly, PgBouncer will in some situations keep around the connection without us being able to close them in the 20 second timeframe we allow the conection termination to take.
While we could try to increase the timeout even further, this is not a good way to guarantee that terminations do indeed connect. Actually, we probably cannot require PgBouncer to really terminate connections only because we kill them in the Postgres backend itself: PgBouncer may reconnect automatically or anything else, so we should treat it as a black box which just does some automagic things we cannot control by issuing commands against Postgres.
Luckily, PgBouncer has an admin console which allows us to take control
over what it does directly. Instead of trying to terminate connections
via Postgres, we can thus ask PgBouncer to terminate connections by
issuing a KILL
command, which will immediately terminate all client
and server connections.
Convert the test cleanup logic to do so. Given that we have test setups
where the same database configuration (and thus database) is reused, we
also need to issue a RESUME
command such that subsequent test cases
will be able to reconnect to the database.