After resetting database, PgBouncer error unexpected response from login query
Summary
After resetting the database, pgbouncer
throws errors and does not allow Sidekiq or Unicorn access to the database.
I am able to connect to the database using gitlab-ctl pgb-console
and with both the pgbouncer
and gitlab-consul
users.
Bypassing pgbouncer
and connecting directly to the database master node allows the application to function correctly.
Steps to reproduce
- Reset the database with
gitlab-rake db:setup
- Start
pgbouncer
and see the error
What is the current bug behavior?
pgbouncer
continuously throws the following error and will not allow Unicorn or Sikekiq access to the database
2019-04-01_15:34:26.85808 2019-04-01 15:34:26.858 720 WARNING C-0x18785d0: gitlabhq_production/(nouser)@1.1.1.1:51312 Pooler Error: unexpected response from login query
2019-04-01_15:34:26.90786 2019-04-01 15:34:26.907 720 LOG S-0x187d410: gitlabhq_production/pgbouncer@2.2.2.2:5432 closing because: unexpected response from login query (age=0)
What is the expected correct behavior?
pgbouncer
should not throw errors and should allow Unicorn and Sidekiq access to the database.
Relevant logs and/or screenshots
2019-04-01_15:34:26.85808 2019-04-01 15:34:26.858 720 WARNING C-0x18785d0: gitlabhq_production/(nouser)@1.1.1.1:51312 Pooler Error: unexpected response from login query
2019-04-01_15:34:26.90786 2019-04-01 15:34:26.907 720 LOG S-0x187d410: gitlabhq_production/pgbouncer@2.2.2.2:5432 closing because: unexpected response from login query (age=0)
Observed in 11.8, although I do not suspect it is version specific. I believe you will see it on all version that uses pgbouncer
version older than 1.9.0
Possible fixes
Upgrade pgbouncer
to 1.9.0
pgbouncer
issue: https://github.com/pgbouncer/pgbouncer/issues/220
'pgbouncer' MR for fix: https://github.com/pgbouncer/pgbouncer/pull/356
We are currently running pgbouncer
1.8.1 and the fix was included in pgbouncer
1.9.0
Customer ticket: https://gitlab.zendesk.com/agent/tickets/117550 (internal use)