PG HA failover didn't reload pgbouncer correctly
https://docs.gitlab.com/ee/administration/high_availability/alpha_database.html#failover-procedure
I stopped the lead database and saw parts of failover happen correctly, but the step of reconfiguring pgbouncer didn't work.
I, [2017-11-13T15:11:51.573994 #14259] INFO -- : rclamp-db-c does not resolve in DNS, using IP address of node instead
I, [2017-11-13T15:11:51.574156 #14259] INFO -- : Found master: 10.133.29.114
I, [2017-11-13T15:11:51.575097 #14259] INFO -- : Running: gitlab-ctl pgb-notify --newhost 10.133.29.114 --user pgbouncer --hostuser gitlab-consul
E, [2017-11-13T15:11:53.363240 #14259] ERROR -- : STDERR: Error running command: GitlabCtl::Errors::ExecutionError
E, [2017-11-13T15:11:53.363428 #14259] ERROR -- : STDERR: ERROR: Password for user pgbouncer:
E, [2017-11-13T15:11:53.363475 #14259] ERROR -- : STDERR: psql: fe_sendauth: no password supplied
E, [2017-11-13T15:11:53.363511 #14259] ERROR -- : STDERR: There was an issue reloading pgbouncer through admin console, sending HUP
E, [2017-11-13T15:11:53.363546 #14259] ERROR -- : STDERR: Unable to reload pgbouncer, restarting instead
E, [2017-11-13T15:11:53.363580 #14259] ERROR -- : STDERR: /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:17:in `rescue in get_command_output': GitlabCtl::Errors::ExecutionError (GitlabCtl::Errors::ExecutionError)
E, [2017-11-13T15:11:53.363614 #14259] ERROR -- : STDERR: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:14:in `get_command_output'
E, [2017-11-13T15:11:53.363648 #14259] ERROR -- : STDERR: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/pgbouncer.rb:116:in `restart'
E, [2017-11-13T15:11:53.363695 #14259] ERROR -- : STDERR: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/pgbouncer.rb:139:in `rescue in notify'
E, [2017-11-13T15:11:53.363741 #14259] ERROR -- : STDERR: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/pgbouncer.rb:135:in `notify'
E, [2017-11-13T15:11:53.363791 #14259] ERROR -- : STDERR: from /opt/gitlab/embedded/service/omnibus-ctl-ee/pgbouncer.rb:27:in `block in load_file'
E, [2017-11-13T15:11:53.363859 #14259] ERROR -- : STDERR: from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.5.0/lib/omnibus-ctl.rb:202:in `block in add_command_under_category'
E, [2017-11-13T15:11:53.363932 #14259] ERROR -- : STDERR: from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.5.0/lib/omnibus-ctl.rb:730:in `run'
E, [2017-11-13T15:11:53.363993 #14259] ERROR -- : STDERR: from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.5.0/bin/omnibus-ctl:31:in `<top (required)>'
E, [2017-11-13T15:11:53.364086 #14259] ERROR -- : STDERR: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
E, [2017-11-13T15:11:53.364129 #14259] ERROR -- : STDERR: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
Maybe caused by: #37 (closed) #41 (closed)
Context: https://gitlab.com/gitlab-org/build/team-tasks/issues/14#note_47036464