pgbouncer fails to configure on new HA setup at restart step, possibly due to missing `databases.ini` file?
Summary
I'm currently working on a new HA GitLab environment and found that the pgbouncer node consistently throws an error every initial reconfigure (clean initial run on a new vm) when it tries to restart:
[0mRecipe: <Dynamically Defined Resource>[0m
* service[pgbouncer] action restart
[0m
================================================================================[0m
[31mError executing action `restart` on resource 'service[pgbouncer]'[0m
================================================================================[0m
[0m Mixlib::ShellOut::ShellCommandFailed[0m
------------------------------------[0m
Expected process to exit with [0], but received '1'
[0m ---- Begin output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer ----
[0m STDOUT: timeout: down: /opt/gitlab/service/pgbouncer: 0s, normally up, want up
[0m STDERR:
[0m ---- End output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer ----
[0m Ran /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer returned 1[0mSubsequent reconfigures appear to pass fine but investigating further I see that pgbouncer is actually still failing, throwing the same error constantly:
2019-09-18_13:17:25.59868 2019-09-18 13:17:25.598 17027 FATAL Cannot load config file
2019-09-18_13:17:26.60251 2019-09-18 13:17:26.602 17028 FATAL Cannot load config file
2019-09-18_13:16:49.45553 2019-09-18 13:16:49.455 16964 FATAL Cannot load config file
2019-09-18_13:16:50.45946 2019-09-18 13:16:50.459 16965 FATAL Cannot load config file
2019-09-18_13:16:51.46310 2019-09-18 13:16:51.463 16966 FATAL Cannot load config file
2019-09-18_13:16:52.46698 2019-09-18 13:16:52.466 16967 FATAL Cannot load config fileDigging around a little more it appears that the default pgbouncer config file references another called databases.ini that should be getting generated on the first install. This doesn't appear to be happening though and it's skipped due to it's not_if condition:
[0m * execute[generate databases.ini] action run (skipped due to not_if)
* execute[generate databases.ini] action nothing (skipped due to action :nothing)Steps to reproduce
Follow the guide here on the official docs - https://docs.gitlab.com/ee/administration/high_availability/pgbouncer.html
What is the current bug behavior?
databases.ini file creation is skipped incorrectly and causes pgbouncer to fail.
What is the expected correct behavior?
databases.ini file is created correctly on a fresh pgbouncer install and pgbouncer starts correctly.
Relevant logs
Relevant logs
[0m * execute[generate databases.ini] action run (skipped due to not_if) * execute[generate databases.ini] action nothing (skipped due to action :nothing) * execute[reload pgbouncer] action nothing (skipped due to action :nothing) * execute[start pgbouncer] action nothing (skipped due to action :nothing) Recipe: <Dynamically Defined Resource>[0m * service[pgbouncer-exporter] action nothing (skipped due to action :nothing) Recipe: gitlab-ee::pgbouncer-exporter_disable[0m * runit_service[pgbouncer-exporter] action disable * ruby_block[disable pgbouncer-exporter] action run (skipped due to only_if) (up to date) Recipe: <Dynamically Defined Resource>[0m * service[repmgrd] action nothing (skipped due to action :nothing) Recipe: repmgr::repmgrd_disable[0m * runit_service[repmgrd] action disable * ruby_block[disable repmgrd] action run (skipped due to only_if) (up to date) Recipe: gitlab-ee::geo-secondary_disable[0m * templatesymlink[Removes database_geo.yml symlink] action delete * file[/var/opt/gitlab/gitlab-rails/etc/database_geo.yml] action delete (up to date) * link[/opt/gitlab/embedded/service/gitlab-rails/config/database_geo.yml] action delete (up to date) (up to date) Recipe: gitlab::gitlab-rails[0m * execute[clear the gitlab-rails cache] action run (skipped due to not_if) Recipe: <Dynamically Defined Resource>[0m * service[node-exporter] action restart [32m- restart service service[node-exporter][0m [0mRecipe: consul::enable_daemon[0m * execute[reload consul] action run [32m- execute /opt/gitlab/bin/gitlab-ctl hup consul[0m [0mRecipe: <Dynamically Defined Resource>[0m * service[pgbouncer] action restart [0m ================================================================================[0m [31mError executing action `restart` on resource 'service[pgbouncer]'[0m ================================================================================[0m [0m Mixlib::ShellOut::ShellCommandFailed[0m ------------------------------------[0m Expected process to exit with [0], but received '1' [0m ---- Begin output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer ---- [0m STDOUT: timeout: down: /opt/gitlab/service/pgbouncer: 1s, normally up, want up [0m STDERR: [0m ---- End output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer ---- [0m Ran /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer returned 1[0m [0m Compiled Resource:[0m ------------------[0m # Declared in [0m [0m service("pgbouncer") do [0m provider Chef::Provider::Service::Simple [0m action [:nothing] [0m default_guard_interpreter :default [0m service_name "pgbouncer" [0m enabled nil [0m running nil [0m masked nil [0m pattern "pgbouncer" [0m start_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv start /opt/gitlab/service/pgbouncer" [0m stop_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv stop /opt/gitlab/service/pgbouncer" [0m status_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv status /opt/gitlab/service/pgbouncer" [0m restart_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer" [0m supports {:restart=>true, :reload=>true, :status=>true} [0m end [0m [0m System Info:[0m ------------[0m chef_version=14.13.11 [0m platform=ubuntu [0m platform_version=18.04 [0m ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux] [0m program_name=/opt/gitlab/embedded/bin/chef-client [0m executable=/opt/gitlab/embedded/bin/chef-client[0m
Details of package version
Provide the package version installation details
Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=========================================-=========================-=========================-======================================================================================= un gitlab-ce <none> <none> (no description available) ii gitlab-ee 12.2.5-ee.0 amd64 GitLab Enterprise Edition (including NGINX, Postgres, Redis)
Environment details
- Operating System:
Ubuntu 18.04 LTS - Installation Target, remove incorrect values:
- VM:
GCP n1-machine-2
- VM:
- Installation Type, remove incorrect values:
- New Installation
- Multinode Install
- Resources
- CPU:
2 Core GCP - Memory total:
8GB
- CPU:
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
# Disable all components except Pgbouncer and Consul agent roles ['pgbouncer_role'] # Configure Pgbouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) # Configure Consul agent consul['watchers'] = %w(postgresql) pgbouncer['users'] = { 'gitlab-consul': { password: 'redacted' }, 'pgbouncer': { password: 'redacted' } } consul['configuration'] = { retry_join: %w(redacted) } node_exporter['listen_address'] = '0.0.0.0:9100' pgbouncer_exporter['listen_address'] = '0.0.0.0:9188'
Edited by Grant Young