failover_pgbouncer can't restart pgbouncer when reloading fails
From /var/log/gitlab/consul/failover_pgbouncer.log:
I, [2018-04-05T00:10:10.687640 #28483] INFO -- : Found master: postgres-01
I, [2018-04-05T00:10:10.688308 #28483] INFO -- : Running: gitlab-ctl pgb-notify --newhost postgres-01 --user pgbouncer --hostuser gitlab-consul
E, [2018-04-05T00:10:11.909444 #28483] ERROR -- : STDERR: Error running command: GitlabCtl::Errors::ExecutionError
E, [2018-04-05T00:10:11.909605 #28483] ERROR -- : STDERR: ERROR: Password for user pgbouncer:
E, [2018-04-05T00:10:11.909666 #28483] ERROR -- : STDERR: psql: fe_sendauth: no password supplied
E, [2018-04-05T00:10:11.909691 #28483] ERROR -- : STDERR: There was an issue reloading pgbouncer through admin console, sending HUP
E, [2018-04-05T00:10:11.909710 #28483] ERROR -- : STDERR: Unable to reload pgbouncer, restarting instead
E, [2018-04-05T00:10:11.909726 #28483] ERROR -- : STDERR: /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:17:in `rescue in get_command_output': GitlabCtl::Errors::ExecutionError (GitlabCtl::Errors::ExecutionError)
E, [2018-04-05T00:10:11.909741 #28483] ERROR -- : STDERR: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:14:in `get_command_output'
E, [2018-04-05T00:10:11.909756 #28483] ERROR -- : STDERR: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/pgbouncer.rb:116:in `restart'
E, [2018-04-05T00:10:11.909770 #28483] ERROR -- : STDERR: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/pgbouncer.rb:139:in `rescue in notify'
E, [2018-04-05T00:10:11.909785 #28483] ERROR -- : STDERR: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/pgbouncer.rb:135:in `notify'
E, [2018-04-05T00:10:11.909800 #28483] ERROR -- : STDERR: from /opt/gitlab/embedded/service/omnibus-ctl-ee/pgbouncer.rb:27:in `block in load_file'
E, [2018-04-05T00:10:11.909820 #28483] ERROR -- : STDERR: from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.5.0/lib/omnibus-ctl.rb:202:in `block in add_command_under_category'
E, [2018-04-05T00:10:11.909835 #28483] ERROR -- : STDERR: from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.5.0/lib/omnibus-ctl.rb:730:in `run'
E, [2018-04-05T00:10:11.909861 #28483] ERROR -- : STDERR: from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.5.0/bin/omnibus-ctl:31:in `<top (required)>'
E, [2018-04-05T00:10:11.909877 #28483] ERROR -- : STDERR: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
E, [2018-04-05T00:10:11.909891 #28483] ERROR -- : STDERR: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
pgbouncer isn't restarted though, instead pgb-notify
got warning: pgbouncer: unable to open supervise/ok: access denied
.