Skip to content

testdb: Fix stuck PgBouncer connections by killing via admin command

Patrick Steinhardt requested to merge pks-pgbouncer-kill-connections into master

The PgBouncer-based test job is really flaky, where the symptom of the flakiness is that we fail to close all connections to the database on test cleanup. Seemingly, PgBouncer will in some situations keep around the connection without us being able to close them in the 20 second timeframe we allow the conection termination to take.

While we could try to increase the timeout even further, this is not a good way to guarantee that terminations do indeed connect. Actually, we probably cannot require PgBouncer to really terminate connections only because we kill them in the Postgres backend itself: PgBouncer may reconnect automatically or anything else, so we should treat it as a black box which just does some automagic things we cannot control by issuing commands against Postgres.

Luckily, PgBouncer has an admin console which allows us to take control over what it does directly. Instead of trying to terminate connections via Postgres, we can thus ask PgBouncer to terminate connections by issuing a KILL command, which will immediately terminate all client and server connections.

Convert the test cleanup logic to do so. Given that we have test setups where the same database configuration (and thus database) is reused, we also need to issue a RESUME command such that subsequent test cases will be able to reconnect to the database.

Edited by Patrick Steinhardt

Merge request reports