PostgresSQL did not respond when Patroni service is enabled
Dear Gitlab users and community members.
I am trying to build a HA GitLab cluster using at on-premise site.
While reconfiguring a postgresql cluster with 3 nodes using https://gitlab.com/gitlab-org/gitlab-environment-toolkit I got a runtime error for Postgres via Patroni
In my_inventory.yml, I have set three postgresql with primary and secondary configuration.
postgres:
hosts:
postgres-1: debian11_with_1.0GB_RAM_1_vCPU_12.0_GB_volume_size
postgres-2: "
postgres-3: "
postgres_primary:
hosts:
postgres-1: "
postgres_secondary:
hosts:
postgres-2: "
postgres-3: "
In vars.yml, 'patroni_password' is activated to utilize patroni.
Common setup is successful and further try to set up postgres in next step.
ansible-playbook -i environments/my_git_envrionment/files/inventory playbooks/common.yml : completed and successful.
ansible-playbook -i environments/my_git_envrionment/files/inventory playbooks/postgres.yml
I am facing the issue on the postgres in this postgres task due to patroni, which is corresponding to the task "gitlab-ctl reconfigure"
Running handlers:
[2022-12-20T14:18:45+00:00] ERROR: Running exception handlers
Running handlers complete
[2022-12-20T14:18:45+00:00] ERROR: Exception handlers complete
Infra Phase failed. 1 resources updated in 01 minutes 48 seconds
[2022-12-20T14:18:45+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2022-12-20T14:18:45+00:00] FATAL: ---------------------------------------------------------------------------------------
[2022-12-20T14:18:45+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2022-12-20T14:18:45+00:00] FATAL: ---------------------------------------------------------------------------------------
[2022-12-20T14:18:45+00:00] FATAL: RuntimeError: ruby_block[wait for postgresql to start] (patroni::enable line 105) had an error:
RuntimeError: PostgreSQL did not respond before service checks were exhausted
stdout_lines: <omitted>
$ gitlab-ctl status
does not show any running postgresql service on the postgres-primary host.
and I have also tried to increase timeout 500 to 700, unfortunately it does not help.
postgresql['max_connections'] = 700
patroni['use_pg_rewind'] = true
patroni['remove_data_directory_on_rewind_failure'] = false
patroni['remove_data_directory_on_diverged_timelines'] = false
It seems that postgres 13.8 is already installed. It can be confirmed by
$ /opt/gitlab/embedded/bin/postgres -V
postgres (PostgreSQL) 13.8
With another view, I can find two versions 12.12 and 13.8 for PostgreSQL
$ sudo find / -wholename '*/bin/postgres' 2>&- | xargs -i xargs -t '{}' -V
/opt/gitlab/embedded/bin/postgres -V
postgres (PostgreSQL) 13.8
/opt/gitlab/embedded/postgresql/13/bin/postgres -V
postgres (PostgreSQL) 13.8
/opt/gitlab/embedded/postgresql/12/bin/postgres -V
postgres (PostgreSQL) 12.12
I cannot perform the start command for postgressql currently. No postgresql can be running at all, I try to run postgresql manually via
$ sudo gitlab-ctl start postgressql
unfortunately, postgresql could not be re-started and therefore reconfiguration could not be completed
$ sudo gitlab-ctl reconfigure
It would be great if you have some guidance on such issue or advice what should I do to resolve this preventing status on the installation of postgresql DB cluster.
Best wishes, Donghee