Bootstrapping an empty patroni cluster will show errors in gitlab-ctl reconfigure
When bootstrapping a new patroni node from a completely empty instance (after removing postgresql data folder), as part of gitlab-ctl reconfigure it tries to restart patroni by issuing a request to patroni's API, but patroni would not have been enabled yet, so there is no API to respond.
Here is the backlog:
[2020-12-03T13:37:54+00:00] INFO: *** Chef Infra Client 15.14.0 ***
[2020-12-03T13:37:54+00:00] INFO: Platform: x86_64-linux
[2020-12-03T13:37:54+00:00] INFO: Chef-client pid: 56330
[2020-12-03T13:37:56+00:00] INFO: Setting the run_list to ["recipe[gitlab-ee]"] from CLI options
[2020-12-03T13:37:56+00:00] INFO: Run List is [recipe[gitlab-ee]]
[2020-12-03T13:37:56+00:00] INFO: Run List expands to [gitlab-ee]
[2020-12-03T13:37:56+00:00] INFO: Starting Chef Infra Client Run for gabriel-patroni-primary-geo-patroni-1.c.group-geo-f9c951.internal
[2020-12-03T13:37:56+00:00] INFO: Running start handlers
[2020-12-03T13:37:56+00:00] INFO: Start handlers complete.
[2020-12-03T13:37:57+00:00] INFO: Loading cookbooks [gitlab-ee@0.0.1, package@0.1.0, gitlab@0.0.1, consul@0.1.0, repmgr@0.1.0, patroni@0.1.0, runit@5.1.3, logrotate@0.0.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, acme@4.1.1, crond@0.1.0]
[2020-12-03T13:37:59+00:00] WARN: Selected systemd because systemctl shows .mount units
[2020-12-03T13:38:00+00:00] WARN: only_if block for templatesymlink[Create a gitlab_pages_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2020-12-03T13:38:00+00:00] WARN: only_if block for templatesymlink[Create a gitlab_kas_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2020-12-03T13:38:01+00:00] INFO: directory[/var/opt/gitlab/patroni/data] created directory /var/opt/gitlab/patroni/data
[2020-12-03T13:38:01+00:00] INFO: directory[/var/opt/gitlab/patroni/data] owner changed to 996
[2020-12-03T13:38:01+00:00] INFO: directory[/var/opt/gitlab/patroni/data] mode changed to 700
[2020-12-03T13:38:01+00:00] INFO: file[/var/opt/gitlab/patroni/data/server.crt] created file /var/opt/gitlab/patroni/data/server.crt
[2020-12-03T13:38:01+00:00] INFO: file[/var/opt/gitlab/patroni/data/server.crt] updated file contents /var/opt/gitlab/patroni/data/server.crt
[2020-12-03T13:38:01+00:00] INFO: file[/var/opt/gitlab/patroni/data/server.crt] owner changed to 996
[2020-12-03T13:38:01+00:00] INFO: file[/var/opt/gitlab/patroni/data/server.crt] group changed to 996
[2020-12-03T13:38:01+00:00] INFO: file[/var/opt/gitlab/patroni/data/server.crt] mode changed to 600
[2020-12-03T13:38:01+00:00] INFO: file[/var/opt/gitlab/patroni/data/server.key] created file /var/opt/gitlab/patroni/data/server.key
[2020-12-03T13:38:01+00:00] INFO: file[/var/opt/gitlab/patroni/data/server.key] updated file contents /var/opt/gitlab/patroni/data/server.key
[2020-12-03T13:38:01+00:00] INFO: file[/var/opt/gitlab/patroni/data/server.key] owner changed to 996
[2020-12-03T13:38:01+00:00] INFO: file[/var/opt/gitlab/patroni/data/server.key] group changed to 996
[2020-12-03T13:38:01+00:00] INFO: file[/var/opt/gitlab/patroni/data/server.key] mode changed to 600
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/postgresql.base.conf] created file /var/opt/gitlab/patroni/data/postgresql.base.conf
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/postgresql.base.conf] updated file contents /var/opt/gitlab/patroni/data/postgresql.base.conf
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/postgresql.base.conf] owner changed to 996
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/postgresql.base.conf] mode changed to 644
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/runtime.conf] created file /var/opt/gitlab/patroni/data/runtime.conf
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/runtime.conf] updated file contents /var/opt/gitlab/patroni/data/runtime.conf
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/runtime.conf] owner changed to 996
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/runtime.conf] mode changed to 644
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/pg_hba.conf] created file /var/opt/gitlab/patroni/data/pg_hba.conf
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/pg_hba.conf] updated file contents /var/opt/gitlab/patroni/data/pg_hba.conf
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/pg_hba.conf] owner changed to 996
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/pg_hba.conf] mode changed to 644
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/pg_ident.conf] created file /var/opt/gitlab/patroni/data/pg_ident.conf
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/pg_ident.conf] updated file contents /var/opt/gitlab/patroni/data/pg_ident.conf
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/pg_ident.conf] owner changed to 996
[2020-12-03T13:38:01+00:00] INFO: template[/var/opt/gitlab/patroni/data/pg_ident.conf] mode changed to 644
[2020-12-03T13:38:02+00:00] INFO: directory[/var/opt/gitlab/patroni] owner changed to 996
[2020-12-03T13:38:02+00:00] INFO: directory[/var/opt/gitlab/patroni] group changed to 996
[2020-12-03T13:38:02+00:00] INFO: directory[/var/opt/gitlab/patroni] mode changed to 700
[2020-12-03T13:38:02+00:00] INFO: directory[/var/opt/gitlab/patroni/data] group changed to 996
[2020-12-03T13:38:02+00:00] INFO: directory[/var/log/gitlab/patroni] created directory /var/log/gitlab/patroni
[2020-12-03T13:38:02+00:00] INFO: directory[/var/log/gitlab/patroni] owner changed to 996
[2020-12-03T13:38:02+00:00] INFO: directory[/var/log/gitlab/patroni] group changed to 996
[2020-12-03T13:38:02+00:00] INFO: directory[/var/log/gitlab/patroni] mode changed to 700
[2020-12-03T13:38:02+00:00] INFO: file[/var/opt/gitlab/patroni/dcs.yaml] created file /var/opt/gitlab/patroni/dcs.yaml
[2020-12-03T13:38:02+00:00] INFO: file[/var/opt/gitlab/patroni/dcs.yaml] updated file contents /var/opt/gitlab/patroni/dcs.yaml
[2020-12-03T13:38:02+00:00] INFO: file[/var/opt/gitlab/patroni/dcs.yaml] owner changed to 996
[2020-12-03T13:38:02+00:00] INFO: file[/var/opt/gitlab/patroni/dcs.yaml] group changed to 996
[2020-12-03T13:38:02+00:00] INFO: file[/var/opt/gitlab/patroni/dcs.yaml] mode changed to 600
[2020-12-03T13:38:02+00:00] INFO: template[/var/opt/gitlab/patroni/patroni.yaml] created file /var/opt/gitlab/patroni/patroni.yaml
[2020-12-03T13:38:02+00:00] INFO: template[/var/opt/gitlab/patroni/patroni.yaml] updated file contents /var/opt/gitlab/patroni/patroni.yaml
[2020-12-03T13:38:02+00:00] INFO: template[/var/opt/gitlab/patroni/patroni.yaml] owner changed to 996
[2020-12-03T13:38:02+00:00] INFO: template[/var/opt/gitlab/patroni/patroni.yaml] group changed to 996
[2020-12-03T13:38:02+00:00] INFO: template[/var/opt/gitlab/patroni/patroni.yaml] mode changed to 600
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni] created directory /opt/gitlab/sv/patroni
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni] owner changed to 0
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni] group changed to 0
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni] mode changed to 755
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/run] created file /opt/gitlab/sv/patroni/run
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/run] updated file contents /opt/gitlab/sv/patroni/run
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/run] owner changed to 0
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/run] group changed to 0
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/run] mode changed to 755
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/log] created directory /opt/gitlab/sv/patroni/log
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/log] owner changed to 0
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/log] group changed to 0
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/log] mode changed to 755
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/log/main] created directory /opt/gitlab/sv/patroni/log/main
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/log/main] owner changed to 0
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/log/main] group changed to 0
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/log/main] mode changed to 755
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/log/config] created file /opt/gitlab/sv/patroni/log/config [89/436]
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/log/config] updated file contents /opt/gitlab/sv/patroni/log/config
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/log/config] owner changed to 0
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/log/config] group changed to 0
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/log/config] mode changed to 644
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/log/config] sending create action to ruby_block[verify_chown_persisted_on_patroni] (immediate)
[2020-12-03T13:38:02+00:00] INFO: ruby_block[verify_chown_persisted_on_patroni] called
[2020-12-03T13:38:02+00:00] INFO: link[/var/log/gitlab/patroni/config] created
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/log/run] created file /opt/gitlab/sv/patroni/log/run
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/log/run] updated file contents /opt/gitlab/sv/patroni/log/run
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/log/run] owner changed to 0
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/log/run] group changed to 0
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/log/run] mode changed to 755
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/env] created directory /opt/gitlab/sv/patroni/env
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/env] owner changed to 0
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/env] group changed to 0
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/env] mode changed to 755
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/control] created directory /opt/gitlab/sv/patroni/control
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/control] owner changed to 0
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/control] group changed to 0
[2020-12-03T13:38:02+00:00] INFO: directory[/opt/gitlab/sv/patroni/control] mode changed to 755
[2020-12-03T13:38:02+00:00] INFO: link[/opt/gitlab/init/patroni] created
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/run] sending run action to ruby_block[restart_service] (delayed)
[2020-12-03T13:38:02+00:00] INFO: template[/opt/gitlab/sv/patroni/log/config] sending create action to ruby_block[reload_log_service] (delayed)
[2020-12-03T13:38:02+00:00] INFO: link[/opt/gitlab/service/patroni] created
[2020-12-03T13:38:05+00:00] INFO: ruby_block[wait for patroni service socket] called
[2020-12-03T13:38:05+00:00] INFO: directory[/opt/gitlab/service/patroni/supervise] mode changed to 755
[2020-12-03T13:38:05+00:00] INFO: directory[/opt/gitlab/service/patroni/log/supervise] mode changed to 755
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/ok] created file /opt/gitlab/service/patroni/supervise/ok
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/ok] owner changed to 996
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/ok] group changed to 996
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/ok] updated atime and mtime to 2020-12-03 13:38:06 +0000
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/ok] created file /opt/gitlab/service/patroni/log/supervise/ok
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/ok] owner changed to 996
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/ok] group changed to 996
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/ok] updated atime and mtime to 2020-12-03 13:38:06 +0000
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/status] owner changed to 996
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/status] group changed to 996
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/status] updated atime and mtime to 2020-12-03 13:38:06 +0000
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/status] owner changed to 996
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/status] group changed to 996
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/status] updated atime and mtime to 2020-12-03 13:38:06 +0000
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/control] created file /opt/gitlab/service/patroni/supervise/control
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/control] owner changed to 996
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/control] group changed to 996
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/control] updated atime and mtime to 2020-12-03 13:38:06 +0000
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/control] created file /opt/gitlab/service/patroni/log/supervise/control
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/control] owner changed to 996
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/control] group changed to 996
[2020-12-03T13:38:06+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/control] updated atime and mtime to 2020-12-03 13:38:06 +0000
[2020-12-03T13:38:07+00:00] INFO: ruby_block[reload_log_service] called
[2020-12-03T13:38:07+00:00] INFO: template[/opt/gitlab/sv/patroni/log/run] sending run action to ruby_block[restart_log_service] (delayed)
[2020-12-03T13:38:07+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/status] owner changed to 996
[2020-12-03T13:38:07+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/status] group changed to 996
[2020-12-03T13:38:07+00:00] INFO: file[/opt/gitlab/service/patroni/supervise/status] updated atime and mtime to 2020-12-03 13:38:07 +0000
[2020-12-03T13:38:07+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/status] owner changed to 996
[2020-12-03T13:38:07+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/status] group changed to 996
[2020-12-03T13:38:07+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/status] updated atime and mtime to 2020-12-03 13:38:07 +0000
[2020-12-03T13:38:07+00:00] INFO: ruby_block[restart_log_service] called
[2020-12-03T13:38:07+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/status] owner changed to 996
[2020-12-03T13:38:07+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/status] group changed to 996
[2020-12-03T13:38:07+00:00] INFO: file[/opt/gitlab/service/patroni/log/supervise/status] updated atime and mtime to 2020-12-03 13:38:07 +0000
[2020-12-03T13:38:07+00:00] INFO: execute[update dynamic configuration settings] running why-run nothing action to support before action
[2020-12-03T13:38:07+00:00] INFO: ruby_block[wait for postgresql to start] called
[2020-12-03T13:38:08+00:00] INFO: execute[signal to restart postgresql] running why-run run action to support before action
[2020-12-03T13:38:09+00:00] INFO: execute[signal to restart postgresql] sending run action to ruby_block[wait for node bootstrap to complete] (before)
[2020-12-03T13:38:10+00:00] INFO: ruby_block[wait for node bootstrap to complete] called
[2020-12-03T13:38:11+00:00] INFO: Running queued delayed notifications before re-raising exception
[2020-12-03T13:38:11+00:00] INFO: file[/var/opt/gitlab/patroni/dcs.yaml] sending run action to execute[update dynamic configuration settings] (delayed)
[2020-12-03T13:38:11+00:00] INFO: execute[update dynamic configuration settings] running why-run run action to support before action
[2020-12-03T13:38:11+00:00] INFO: execute[update dynamic configuration settings] sending run action to ruby_block[wait for node bootstrap to complete] (before)
[2020-12-03T13:38:11+00:00] INFO: ruby_block[wait for node bootstrap to complete] called
[2020-12-03T13:38:12+00:00] INFO: execute[update dynamic configuration settings] ran successfully
[2020-12-03T13:38:12+00:00] ERROR: Running exception handlers
[2020-12-03T13:38:12+00:00] ERROR: Exception handlers complete
[2020-12-03T13:38:12+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2020-12-03T13:38:12+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-12-03T13:38:12+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: execute[signal to restart postgresql] (patroni::enable line 124) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/patronictl -c /var/opt/gitlab/patroni/patroni.yaml restart --force postgresql-ha gabriel-patroni-primary-geo-patroni-1.c.group-geo-f9c951.internal ----
STDOUT: +---------------+-------------------------------------------------------------------+-------------+--------+---------+----+-----------+
| Cluster | Member | Host | Role | State | TL | Lag in MB |
+---------------+-------------------------------------------------------------------+-------------+--------+---------+----+-----------+
| postgresql-ha | gabriel-patroni-primary-geo-patroni-1.c.group-geo-f9c951.internal | 10.164.0.19 | | running | 1 | 41 |
| postgresql-ha | gabriel-patroni-primary-geo-patroni-2.c.group-geo-f9c951.internal | 10.164.0.20 | | running | 1 | 41 |
| postgresql-ha | gabriel-patroni-primary-geo-patroni-3.c.group-geo-f9c951.internal | 10.164.0.21 | | running | 1 | 41 |
| postgresql-ha | gabriel-patroni-primary-geo.c.group-geo-f9c951.internal | 10.164.0.14 | Leader | running | 2 | |
+---------------+-------------------------------------------------------------------+-------------+--------+---------+----+-----------+
STDERR: 2020-12-03 13:38:10,981 - WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9290b5a390>: Failed to establish a new connection: [Errno 111] Connection refused')': /restart
2020-12-03 13:38:10,982 - WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9290b5a510>: Failed to establish a new connection: [Errno 111] Connection refused')': /restart
2020-12-03 13:38:10,983 - WARNING - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9290b5aa50>: Failed to establish a new connection: [Errno 111] Connection refused')': /restart
Traceback (most recent call last):
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connection.py", line 170, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/util/connection.py", line 96, in create_connection
raise err
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/util/connection.py", line 86, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1277, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1323, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1272, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1032, in _send_output
self.send(msg)
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 972, in send
self.connect()
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connection.py", line 200, in connect
conn = self._new_conn()
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f9290b5ae90>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1277, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1323, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1272, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1032, in _send_output
self.send(msg)
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 972, in send
self.connect()
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connection.py", line 200, in connect
conn = self._new_conn()
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connection.py", line 182, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f9290b5ae90>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/gitlab/embedded/bin/patronictl", line 8, in <module>
sys.exit(ctl())
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/ctl.py", line 544, in restart
r = request_patroni(member, 'post', 'restart', content)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/ctl.py", line 133, in request_patroni
return request_executor(member, method, endpoint, data)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/request.py", line 53, in __call__
return self.request(method, url, data, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/request.py", line 46, in request
return self._pool.request(method.upper(), url, body=body, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/request.py", line 79, in request
method, url, fields=fields, headers=headers, **urlopen_kw
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/request.py", line 170, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/poolmanager.py", line 375, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 796, in urlopen
**response_kw
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 796, in urlopen
**response_kw
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 796, in urlopen
**response_kw
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/util/retry.py", line 573, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.164.0.19', port=8008): Max retries exceeded with url: /restart (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9290b5ae90>: Failed to establish a new connection: [Errno 111] Connection refused'))
---- End output of /opt/gitlab/embedded/bin/patronictl -c /var/opt/gitlab/patroni/patroni.yaml restart --force postgresql-ha gabriel-patroni-primary-geo-patroni-1.c.group-geo-f9c951.internal ----
Ran /opt/gitlab/embedded/bin/patronictl -c /var/opt/gitlab/patroni/patroni.yaml restart --force postgresql-ha gabriel-patroni-primary-geo-patroni-1.c.group-geo-f9c951.internal returned 1as a side-effect we have both patroni and postgresql running:
ok: run: consul: (pid 108380) 0s
ok: run: logrotate: (pid 108391) 1s
ok: run: node-exporter: (pid 108418) 0s
ok: run: patroni: (pid 108424) 1s
ok: run: postgres-exporter: (pid 108443) 0s
ok: run: postgresql: (pid 108449) 1sEdited by Michael Kozono