pgbouncer reload failure should provide insight

pgbouncer reload failure should provide insight

If a pgbouncer reload fails, provide the administrator with a helpful
message that links to the documentation for how to fix it instead of
trying a HUP and restart which will never actually work.

Closes https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3380

Changelog: fixed

What does this MR do?

The consul user is unable to execute either of the fall-back commands for reloading PgBouncer - hup or restart

root@gitlab-testdbc1-bouncer1:/# su gitlab-consul
$ id
uid=991(gitlab-consul) gid=991(gitlab-consul) groups=991(gitlab-consul)
$ gitlab-ctl hup pgbouncer
warning: pgbouncer: unable to open supervise/ok: access denied
$ gitlab-ctl restart pgbouncer
warning: pgbouncer: unable to open supervise/ok: access denied

Rather than trying, and failing, and then logging the following (and more) in /var/log/gitlab/consul/failover_pgbouncer.log let's log how to fix it.

I, [2022-05-30T17:47:16.072981 #583]  INFO -- : Running: gitlab-ctl pgb-notify --newhost hostname --user pgbouncer --hostuser gitlab-consul
E, [2022-05-30T17:47:25.511963 #583] ERROR -- : STDERR: Error running command: GitlabCtl::Errors::ExecutionError

E, [2022-05-30T17:47:25.512052 #583] ERROR -- : STDERR: ERROR: Password for user pgbouncer: 

E, [2022-05-30T17:47:25.512088 #583] ERROR -- : STDERR: psql: error: fe_sendauth: no password supplied

E, [2022-05-30T17:47:25.512117 #583] ERROR -- : STDERR: Error running command: GitlabCtl::Errors::ExecutionError

E, [2022-05-30T17:47:25.512155 #583] ERROR -- : STDERR: ERROR: Password for user pgbouncer: 

E, [2022-05-30T17:47:25.512190 #583] ERROR -- : STDERR: psql: error: fe_sendauth: no password supplied

E, [2022-05-30T17:47:25.512218 #583] ERROR -- : STDERR: There was an issue reloading pgbouncer through admin console, sending HUP

E, [2022-05-30T17:47:25.512253 #583] ERROR -- : STDERR: Unable to reload pgbouncer, restarting instead

E, [2022-05-30T17:47:25.512287 #583] ERROR -- : STDERR: /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:16:in `rescue in get_command_output': GitlabCtl::Errors::ExecutionError (GitlabCtl::Errors::ExecutionError)

E, [2022-05-30T17:47:25.512321 #583] ERROR -- : STDERR: 	from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:12:in `get_command_output'

E, [2022-05-30T17:47:25.512354 #583] ERROR -- : STDERR: 	from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/pgbouncer.rb:142:in `restart'

E, [2022-05-30T17:47:25.512408 #583] ERROR -- : STDERR: 	from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/pgbouncer.rb:167:in `rescue in notify'

E, [2022-05-30T17:47:25.512437 #583] ERROR -- : STDERR: 	from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/pgbouncer.rb:163:in `notify'

E, [2022-05-30T17:47:25.512473 #583] ERROR -- : STDERR: 	from /opt/gitlab/embedded/service/omnibus-ctl-ee/pgbouncer.rb:27:in `block in load_file'

E, [2022-05-30T17:47:25.512501 #583] ERROR -- : STDERR: 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'

E, [2022-05-30T17:47:25.512571 #583] ERROR -- : STDERR: 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'

E, [2022-05-30T17:47:25.512602 #583] ERROR -- : STDERR: 	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'

E, [2022-05-30T17:47:25.512629 #583] ERROR -- : STDERR: 	from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'

E, [2022-05-30T17:47:25.512655 #583] ERROR -- : STDERR: 	from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'

E, [2022-05-30T17:47:25.512681 #583] ERROR -- : STDERR: /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/mixlib-shellout-3.2.5/lib/mixlib/shellout.rb:300:in `invalid!': Expected process to exit with [0], but received '1' (Mixlib::ShellOut::ShellCommandFailed)

<snip>

As proposed:

I, [2022-05-30T18:13:15.619775 #1047]  INFO -- : Running: gitlab-ctl pgb-notify --newhost hostname --user pgbouncer --hostuser gitlab-consul
E, [2022-05-30T18:13:18.654720 #1047] ERROR -- : STDERR: Error running command: GitlabCtl::Errors::ExecutionError

E, [2022-05-30T18:13:18.654809 #1047] ERROR -- : STDERR: ERROR: Password for user pgbouncer: 

E, [2022-05-30T18:13:18.654844 #1047] ERROR -- : STDERR: psql: error: fe_sendauth: no password supplied

E, [2022-05-30T18:13:18.654871 #1047] ERROR -- : STDERR: Error running command: GitlabCtl::Errors::ExecutionError

E, [2022-05-30T18:13:18.654897 #1047] ERROR -- : STDERR: ERROR: Password for user pgbouncer: 

E, [2022-05-30T18:13:18.654922 #1047] ERROR -- : STDERR: psql: error: fe_sendauth: no password supplied

E, [2022-05-30T18:13:18.654946 #1047] ERROR -- : STDERR: There was an issue reloading pgbouncer through admin console

E, [2022-05-30T18:13:18.654972 #1047] ERROR -- : STDERR: Check that gitlab-ctl write-pgpass has been run to populate ~/.pgpass for the Consul account

Related issues

#3380 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

• Merge Request Title, and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• Pipeline is green on the dev.gitlab.org mirror for this branch if the change is touching anything besides documentation or internal cookbooks. Please note that feature branches are not automatically mirrored to dev.gitlab.org, and should be manually pushed whenever necessary.
• trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for the GitLab Chart opened
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.