### Summary I'm currently working on a new HA GitLab environment and found that the pgbouncer node consistently throws an error every initial reconfigure (clean initial run on a new vm) when it tries to restart: ``` [0mRecipe: <Dynamically Defined Resource>[0m * service[pgbouncer] action restart [0m ================================================================================[0m [31mError executing action `restart` on resource 'service[pgbouncer]'[0m ================================================================================[0m [0m Mixlib::ShellOut::ShellCommandFailed[0m ------------------------------------[0m Expected process to exit with [0], but received '1' [0m ---- Begin output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer ---- [0m STDOUT: timeout: down: /opt/gitlab/service/pgbouncer: 0s, normally up, want up [0m STDERR: [0m ---- End output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer ---- [0m Ran /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer returned 1[0m ``` Subsequent reconfigures appear to pass fine but investigating further I see that pgbouncer is actually still failing, throwing the same error constantly: ``` 2019-09-18_13:17:25.59868 2019-09-18 13:17:25.598 17027 FATAL Cannot load config file 2019-09-18_13:17:26.60251 2019-09-18 13:17:26.602 17028 FATAL Cannot load config file 2019-09-18_13:16:49.45553 2019-09-18 13:16:49.455 16964 FATAL Cannot load config file 2019-09-18_13:16:50.45946 2019-09-18 13:16:50.459 16965 FATAL Cannot load config file 2019-09-18_13:16:51.46310 2019-09-18 13:16:51.463 16966 FATAL Cannot load config file 2019-09-18_13:16:52.46698 2019-09-18 13:16:52.466 16967 FATAL Cannot load config file ``` Digging around a little more it appears that the default pgbouncer config file references another called databases.ini that [should be getting generated on the first install](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab-ee/recipes/pgbouncer.rb#L78). This doesn't appear to be happening though and it's skipped due to it's not_if condition: ``` [0m * execute[generate databases.ini] action run (skipped due to not_if) * execute[generate databases.ini] action nothing (skipped due to action :nothing) ``` ### Steps to reproduce Follow the guide here on the official docs - https://docs.gitlab.com/ee/administration/high_availability/pgbouncer.html ### What is the current *bug* behavior? `databases.ini` file creation is skipped incorrectly and causes pgbouncer to fail. ### What is the expected *correct* behavior? `databases.ini` file is created correctly on a fresh pgbouncer install and pgbouncer starts correctly. ### Relevant logs <details> <summary> Relevant logs </summary> <pre> ``` [0m * execute[generate databases.ini] action run (skipped due to not_if) * execute[generate databases.ini] action nothing (skipped due to action :nothing) * execute[reload pgbouncer] action nothing (skipped due to action :nothing) * execute[start pgbouncer] action nothing (skipped due to action :nothing) Recipe: <Dynamically Defined Resource>[0m * service[pgbouncer-exporter] action nothing (skipped due to action :nothing) Recipe: gitlab-ee::pgbouncer-exporter_disable[0m * runit_service[pgbouncer-exporter] action disable * ruby_block[disable pgbouncer-exporter] action run (skipped due to only_if) (up to date) Recipe: <Dynamically Defined Resource>[0m * service[repmgrd] action nothing (skipped due to action :nothing) Recipe: repmgr::repmgrd_disable[0m * runit_service[repmgrd] action disable * ruby_block[disable repmgrd] action run (skipped due to only_if) (up to date) Recipe: gitlab-ee::geo-secondary_disable[0m * templatesymlink[Removes database_geo.yml symlink] action delete * file[/var/opt/gitlab/gitlab-rails/etc/database_geo.yml] action delete (up to date) * link[/opt/gitlab/embedded/service/gitlab-rails/config/database_geo.yml] action delete (up to date) (up to date) Recipe: gitlab::gitlab-rails[0m * execute[clear the gitlab-rails cache] action run (skipped due to not_if) Recipe: <Dynamically Defined Resource>[0m * service[node-exporter] action restart [32m- restart service service[node-exporter][0m [0mRecipe: consul::enable_daemon[0m * execute[reload consul] action run [32m- execute /opt/gitlab/bin/gitlab-ctl hup consul[0m [0mRecipe: <Dynamically Defined Resource>[0m * service[pgbouncer] action restart [0m ================================================================================[0m [31mError executing action `restart` on resource 'service[pgbouncer]'[0m ================================================================================[0m [0m Mixlib::ShellOut::ShellCommandFailed[0m ------------------------------------[0m Expected process to exit with [0], but received '1' [0m ---- Begin output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer ---- [0m STDOUT: timeout: down: /opt/gitlab/service/pgbouncer: 1s, normally up, want up [0m STDERR: [0m ---- End output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer ---- [0m Ran /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer returned 1[0m [0m Compiled Resource:[0m ------------------[0m # Declared in [0m [0m service("pgbouncer") do [0m provider Chef::Provider::Service::Simple [0m action [:nothing] [0m default_guard_interpreter :default [0m service_name "pgbouncer" [0m enabled nil [0m running nil [0m masked nil [0m pattern "pgbouncer" [0m start_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv start /opt/gitlab/service/pgbouncer" [0m stop_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv stop /opt/gitlab/service/pgbouncer" [0m status_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv status /opt/gitlab/service/pgbouncer" [0m restart_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/pgbouncer" [0m supports {:restart=>true, :reload=>true, :status=>true} [0m end [0m [0m System Info:[0m ------------[0m chef_version=14.13.11 [0m platform=ubuntu [0m platform_version=18.04 [0m ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux] [0m program_name=/opt/gitlab/embedded/bin/chef-client [0m executable=/opt/gitlab/embedded/bin/chef-client[0m ``` </pre> </details> ### Details of package version <details> <summary>Provide the package version installation details</summary> <pre> ``` Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=========================================-=========================-=========================-======================================================================================= un gitlab-ce <none> <none> (no description available) ii gitlab-ee 12.2.5-ee.0 amd64 GitLab Enterprise Edition (including NGINX, Postgres, Redis) ``` </pre> </details> ### Environment details * Operating System: `Ubuntu 18.04 LTS` * Installation Target, remove incorrect values: * VM: `GCP n1-machine-2` * Installation Type, remove incorrect values: * New Installation * Multinode Install * Resources * CPU: `2 Core GCP` * Memory total: `8GB` ### Configuration details <details> <summary> Provide the relevant sections of `/etc/gitlab/gitlab.rb` </summary> <pre> ``` # Disable all components except Pgbouncer and Consul agent roles ['pgbouncer_role'] # Configure Pgbouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) # Configure Consul agent consul['watchers'] = %w(postgresql) pgbouncer['users'] = { 'gitlab-consul': { password: 'redacted' }, 'pgbouncer': { password: 'redacted' } } consul['configuration'] = { retry_join: %w(redacted) } node_exporter['listen_address'] = '0.0.0.0:9100' pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' ``` </pre> </details>