After resetting database, PgBouncer error unexpected response from login query
Summary
After resetting the database, pgbouncer throws errors and does not allow Sidekiq or Unicorn access to the database.
I am able to connect to the database using gitlab-ctl pgb-console and with both the pgbouncer and gitlab-consul users.
Bypassing pgbouncer and connecting directly to the database master node allows the application to function correctly.
Steps to reproduce
- Reset the database with
gitlab-rake db:setup - Start
pgbouncerand see the error
What is the current bug behavior?
pgbouncer continuously throws the following error and will not allow Unicorn or Sikekiq access to the database
2019-04-01_15:34:26.85808 2019-04-01 15:34:26.858 720 WARNING C-0x18785d0: gitlabhq_production/(nouser)@1.1.1.1:51312 Pooler Error: unexpected response from login query
2019-04-01_15:34:26.90786 2019-04-01 15:34:26.907 720 LOG S-0x187d410: gitlabhq_production/pgbouncer@2.2.2.2:5432 closing because: unexpected response from login query (age=0)What is the expected correct behavior?
pgbouncer should not throw errors and should allow Unicorn and Sidekiq access to the database.
Relevant logs and/or screenshots
2019-04-01_15:34:26.85808 2019-04-01 15:34:26.858 720 WARNING C-0x18785d0: gitlabhq_production/(nouser)@1.1.1.1:51312 Pooler Error: unexpected response from login query
2019-04-01_15:34:26.90786 2019-04-01 15:34:26.907 720 LOG S-0x187d410: gitlabhq_production/pgbouncer@2.2.2.2:5432 closing because: unexpected response from login query (age=0)Observed in 11.8, although I do not suspect it is version specific. I believe you will see it on all version that uses pgbouncer version older than 1.9.0
Possible fixes
Upgrade pgbouncer to 1.9.0
pgbouncer issue: https://github.com/pgbouncer/pgbouncer/issues/220
'pgbouncer' MR for fix: https://github.com/pgbouncer/pgbouncer/pull/356
We are currently running pgbouncer 1.8.1 and the fix was included in pgbouncer 1.9.0
Customer ticket: https://gitlab.zendesk.com/agent/tickets/117550 (internal use)