Geo - The gitlab-ctl reconfigure fails when enabling a Patroni standby cluster on a secondary node
Summary
While spinning up a new Geo deployment with Patroni, the gitlab-ctl reconfigure command fails after enabling the Patroni standby cluster on a Geo secondary node.
Relevant logs
Recipe: consul::enable
* account[Consul user and group] action create
* group[Consul user and group] action create (up to date)
* linux_user[Consul user and group] action create (up to date)
(up to date)
* directory[/var/opt/gitlab/consul] action create (up to date)
* directory[/opt/gitlab/etc/consul/env] action create (up to date)
* env_dir[/opt/gitlab/etc/consul/env] action create
* directory[/opt/gitlab/etc/consul/env] action create (up to date)
* file[/opt/gitlab/etc/consul/env/SSL_CERT_DIR] action create (up to date)
(up to date)
* directory[/var/opt/gitlab/consul/config.d] action create (up to date)
* directory[/var/opt/gitlab/consul/data] action create (up to date)
* directory[/var/log/gitlab/consul] action create (up to date)
* directory[/var/opt/gitlab/consul/scripts] action create (up to date)
* file[/var/opt/gitlab/consul/config.json] action create (up to date)
* ruby_block[consul config change] action nothing (skipped due to action :nothing)
Recipe: consul::enable_service_postgresql
* file[/var/opt/gitlab/consul/config.d/postgresql_service.json] action create
- update content in file /var/opt/gitlab/consul/config.d/postgresql_service.json from 470b1f to 5afe9c
--- /var/opt/gitlab/consul/config.d/postgresql_service.json 2021-01-26 17:55:41.908968054 +0000
+++ /var/opt/gitlab/consul/config.d/.chef-postgresql_service20210126-38123-1wewn25.json 2021-01-26 18:20:14.746095692 +0000
@@ -1 +1 @@
-{"service":{"name":"postgresql","address":"","port":5432,"check":{"id":"service:postgresql","interval":"10s","status":"failing","args":["/opt/gitlab/bin/gitlab-ctl","repmgr-check-master"]}},"watches":[{"type":"keyprefix","prefix":"gitlab/ha/postgresql/failed_masters/","args":["/opt/gitlab/bin/gitlab-ctl","consul","watchers","handle-failed-master"]}]}
+{"service":{"name":"postgresql","address":"","port":5432,"check":{"id":"service:postgresql","interval":"10s","status":"failing","args":["/opt/gitlab/bin/gitlab-ctl","patroni","check-standby-leader"]}}}
Recipe: consul::enable_daemon
* service[consul] action nothing (skipped due to action :nothing)
* runit_service[consul] action enable
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/consul] action create (up to date)
* template[/opt/gitlab/sv/consul/run] action create (up to date)
* directory[/opt/gitlab/sv/consul/log] action create (up to date)
* directory[/opt/gitlab/sv/consul/log/main] action create (up to date)
* template[/opt/gitlab/sv/consul/log/config] action create (up to date)
* ruby_block[verify_chown_persisted_on_consul] action nothing (skipped due to action :nothing)
* link[/var/log/gitlab/consul/config] action create (up to date)
* template[/opt/gitlab/sv/consul/log/run] action create (up to date)
* directory[/opt/gitlab/sv/consul/env] action create (up to date)
* ruby_block[Delete unmanaged env files for consul service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/consul/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/consul/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/consul/control] action create (up to date)
* link[/opt/gitlab/init/consul] action create (up to date)
* file[/opt/gitlab/sv/consul/down] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/consul] action create (up to date)
* ruby_block[wait for consul service socket] action run (skipped due to not_if)
* directory[/opt/gitlab/service/consul/supervise] action create (up to date)
* directory[/opt/gitlab/service/consul/log/supervise] action create (up to date)
* file[/opt/gitlab/service/consul/supervise/ok] action touch (skipped due to only_if)
* file[/opt/gitlab/service/consul/log/supervise/ok] action touch (skipped due to only_if)
* file[/opt/gitlab/service/consul/supervise/status] action touch
- change owner from 'root' to 'gitlab-consul'
- change group from 'root' to 'gitlab-consul'
- update utime on file /opt/gitlab/service/consul/supervise/status
* file[/opt/gitlab/service/consul/log/supervise/status] action touch (skipped due to only_if)
* file[/opt/gitlab/service/consul/supervise/control] action touch (skipped due to only_if)
* file[/opt/gitlab/service/consul/log/supervise/control] action touch (skipped due to only_if)
* execute[reload consul] action nothing (skipped due to action :nothing)
Recipe: repmgr::repmgrd_disable
* service[repmgrd] action nothing (skipped due to action :nothing)
* runit_service[repmgrd] action disable
* ruby_block[disable repmgrd] action run (skipped due to only_if)
(up to date)
Recipe: patroni::enable
* directory[/var/opt/gitlab/patroni] action create
- change mode from '0755' to '0700'
- change owner from 'root' to 'gitlab-psql'
- change group from 'root' to 'gitlab-psql'
* directory[/var/opt/gitlab/patroni/data] action create
- change group from 'root' to 'gitlab-psql'
* directory[/var/log/gitlab/patroni] action create
- create new directory /var/log/gitlab/patroni
- change mode from '' to '0700'
- change owner from '' to 'gitlab-psql'
- change group from '' to 'gitlab-psql'
* file[/var/opt/gitlab/patroni/dcs.yaml] action create
- create new file /var/opt/gitlab/patroni/dcs.yaml
- update content in file /var/opt/gitlab/patroni/dcs.yaml from none to fb040b
--- /var/opt/gitlab/patroni/dcs.yaml 2021-01-26 18:20:14.906095383 +0000
+++ /var/opt/gitlab/patroni/.chef-dcs20210126-38123-1oc0lna.yaml 2021-01-26 18:20:14.906095383 +0000
@@ -1,28 +1,56 @@
+---
+postgresql:
+ parameters:
+ wal_level: replica
+ hot_standby: 'on'
+ wal_keep_segments: 10
+ max_wal_senders: 10
+ max_replication_slots: 10
+ checkpoint_timeout: 30
+ max_prepared_transactions: 0
+ track_commit_timestamp: 'off'
+ max_connections: 200
+ max_locks_per_transaction: 128
+ max_worker_processes: 8
+ wal_log_hints: 'on'
+ use_pg_rewind: true
+ use_slots: true
+slots: {}
+loop_wait: 10
+ttl: 30
+retry_timeout: 10
+maximum_lag_on_failover: 1048576
+max_timelines_history: 0
+master_start_timeout: 300
+standby_cluster:
+ host: 10.164.0.19
+ port: 5432
+ primary_slot_name: geo_secondary
- change mode from '' to '0600'
- change owner from '' to 'gitlab-psql'
- change group from '' to 'gitlab-psql'
* template[/var/opt/gitlab/patroni/patroni.yaml] action create
- create new file /var/opt/gitlab/patroni/patroni.yaml
- update content in file /var/opt/gitlab/patroni/patroni.yaml from none to b1cb75
- suppressed sensitive resource
- change mode from '' to '0600'
- change owner from '' to 'gitlab-psql'
- change group from '' to 'gitlab-psql'
* service[patroni] action nothing (skipped due to action :nothing)
* runit_service[patroni] action enable
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/patroni] action create
- create new directory /opt/gitlab/sv/patroni
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* template[/opt/gitlab/sv/patroni/run] action create
- create new file /opt/gitlab/sv/patroni/run
- update content in file /opt/gitlab/sv/patroni/run from none to 789ffa
--- /opt/gitlab/sv/patroni/run 2021-01-26 18:20:14.974095250 +0000
+++ /opt/gitlab/sv/patroni/.chef-run20210126-38123-1o4l34u 2021-01-26 18:20:14.974095250 +0000
@@ -1,4 +1,8 @@
+#!/bin/sh
+exec 2>&1
+
+exec chpst -P -U gitlab-psql:gitlab-psql -u gitlab-psql:gitlab-psql /opt/gitlab/embedded/bin/patroni /var/opt/gitlab/patroni/patroni.yaml
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* directory[/opt/gitlab/sv/patroni/log] action create
- create new directory /opt/gitlab/sv/patroni/log
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* directory[/opt/gitlab/sv/patroni/log/main] action create
- create new directory /opt/gitlab/sv/patroni/log/main
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* template[/opt/gitlab/sv/patroni/log/config] action create
- create new file /opt/gitlab/sv/patroni/log/config
- update content in file /opt/gitlab/sv/patroni/log/config from none to 623c00
--- /opt/gitlab/sv/patroni/log/config 2021-01-26 18:20:15.002095197 +0000
+++ /opt/gitlab/sv/patroni/log/.chef-config20210126-38123-9rge23 2021-01-26 18:20:15.002095197 +0000
@@ -1,6 +1,12 @@
+s209715200
+n30
+t86400
+!gzip
+
+
- change mode from '' to '0644'
- change owner from '' to 'root'
- change group from '' to 'root'
* ruby_block[verify_chown_persisted_on_patroni] action create
- execute the ruby block verify_chown_persisted_on_patroni
* ruby_block[verify_chown_persisted_on_patroni] action nothing (skipped due to action :nothing)
* link[/var/log/gitlab/patroni/config] action create
- create symlink at /var/log/gitlab/patroni/config to /opt/gitlab/sv/patroni/log/config
* template[/opt/gitlab/sv/patroni/log/run] action create
- create new file /opt/gitlab/sv/patroni/log/run
- update content in file /opt/gitlab/sv/patroni/log/run from none to b2278f
--- /opt/gitlab/sv/patroni/log/run 2021-01-26 18:20:15.030095143 +0000
+++ /opt/gitlab/sv/patroni/log/.chef-run20210126-38123-16gbcpl 2021-01-26 18:20:15.030095143 +0000
@@ -1,2 +1,4 @@
+#!/bin/sh
+exec svlogd -tt /var/log/gitlab/patroni
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* directory[/opt/gitlab/sv/patroni/env] action create
- create new directory /opt/gitlab/sv/patroni/env
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* ruby_block[Delete unmanaged env files for patroni service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/patroni/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/patroni/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/patroni/control] action create
- create new directory /opt/gitlab/sv/patroni/control
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* link[/opt/gitlab/init/patroni] action create
- create symlink at /opt/gitlab/init/patroni to /opt/gitlab/embedded/bin/sv
* file[/opt/gitlab/sv/patroni/down] action nothing (skipped due to action :nothing)
* ruby_block[restart_service] action run (skipped due to only_if)
* ruby_block[reload_log_service] action create
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/patroni] action create (up to date)
* template[/opt/gitlab/sv/patroni/run] action create (up to date)
* directory[/opt/gitlab/sv/patroni/log] action create (up to date)
* directory[/opt/gitlab/sv/patroni/log/main] action create (up to date)
* template[/opt/gitlab/sv/patroni/log/config] action create (up to date)
* ruby_block[verify_chown_persisted_on_patroni] action nothing (skipped due to action :nothing)
* link[/var/log/gitlab/patroni/config] action create (up to date)
* template[/opt/gitlab/sv/patroni/log/run] action create (up to date)
* directory[/opt/gitlab/sv/patroni/env] action create (up to date)
* ruby_block[Delete unmanaged env files for patroni service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/patroni/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/patroni/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/patroni/control] action create (up to date)
* link[/opt/gitlab/init/patroni] action create (up to date)
* file[/opt/gitlab/sv/patroni/down] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/patroni] action create
- create symlink at /opt/gitlab/service/patroni to /opt/gitlab/sv/patroni
* ruby_block[wait for patroni service socket] action run
- execute the ruby block wait for patroni service socket
* directory[/opt/gitlab/service/patroni/supervise] action create
- change mode from '0700' to '0755'
* directory[/opt/gitlab/service/patroni/log/supervise] action create
- change mode from '0700' to '0755'
* file[/opt/gitlab/service/patroni/supervise/ok] action touch
- create new file /opt/gitlab/service/patroni/supervise/ok
- change owner from '' to 'gitlab-psql'
- change group from '' to 'gitlab-psql'
- update utime on file /opt/gitlab/service/patroni/supervise/ok
* file[/opt/gitlab/service/patroni/log/supervise/ok] action touch
- create new file /opt/gitlab/service/patroni/log/supervise/ok
- change owner from '' to 'gitlab-psql'
- change group from '' to 'gitlab-psql'
- update utime on file /opt/gitlab/service/patroni/log/supervise/ok
* file[/opt/gitlab/service/patroni/supervise/status] action touch
- change owner from 'root' to 'gitlab-psql'
- change group from 'root' to 'gitlab-psql'
- update utime on file /opt/gitlab/service/patroni/supervise/status
* file[/opt/gitlab/service/patroni/log/supervise/status] action touch
- change owner from 'root' to 'gitlab-psql'
- change group from 'root' to 'gitlab-psql'
- update utime on file /opt/gitlab/service/patroni/log/supervise/status
* file[/opt/gitlab/service/patroni/supervise/control] action touch
- create new file /opt/gitlab/service/patroni/supervise/control
- change owner from '' to 'gitlab-psql'
- change group from '' to 'gitlab-psql'
- update utime on file /opt/gitlab/service/patroni/supervise/control
* file[/opt/gitlab/service/patroni/log/supervise/control] action touch
- create new file /opt/gitlab/service/patroni/log/supervise/control
- change owner from '' to 'gitlab-psql'
- change group from '' to 'gitlab-psql'
- update utime on file /opt/gitlab/service/patroni/log/supervise/control
- execute the ruby block reload_log_service
* ruby_block[restart_log_service] action run
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/patroni] action create (up to date)
* template[/opt/gitlab/sv/patroni/run] action create (up to date)
* directory[/opt/gitlab/sv/patroni/log] action create (up to date)
* directory[/opt/gitlab/sv/patroni/log/main] action create (up to date)
* template[/opt/gitlab/sv/patroni/log/config] action create (up to date)
* ruby_block[verify_chown_persisted_on_patroni] action nothing (skipped due to action :nothing)
* link[/var/log/gitlab/patroni/config] action create (up to date)
* template[/opt/gitlab/sv/patroni/log/run] action create (up to date)
* directory[/opt/gitlab/sv/patroni/env] action create (up to date)
* ruby_block[Delete unmanaged env files for patroni service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/patroni/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/patroni/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/patroni/control] action create (up to date)
* link[/opt/gitlab/init/patroni] action create (up to date)
* file[/opt/gitlab/sv/patroni/down] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/patroni] action create (up to date)
* ruby_block[wait for patroni service socket] action run (skipped due to not_if)
* directory[/opt/gitlab/service/patroni/supervise] action create (up to date)
* directory[/opt/gitlab/service/patroni/log/supervise] action create (up to date)
* file[/opt/gitlab/service/patroni/supervise/ok] action touch (skipped due to only_if)
* file[/opt/gitlab/service/patroni/log/supervise/ok] action touch (skipped due to only_if)
* file[/opt/gitlab/service/patroni/supervise/status] action touch
- change owner from 'root' to 'gitlab-psql'
- change group from 'root' to 'gitlab-psql'
- update utime on file /opt/gitlab/service/patroni/supervise/status
* file[/opt/gitlab/service/patroni/log/supervise/status] action touch
- change owner from 'root' to 'gitlab-psql'
- change group from 'root' to 'gitlab-psql'
- update utime on file /opt/gitlab/service/patroni/log/supervise/status
* file[/opt/gitlab/service/patroni/supervise/control] action touch (skipped due to only_if)
* file[/opt/gitlab/service/patroni/log/supervise/control] action touch (skipped due to only_if)
- execute the ruby block restart_log_service
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/patroni] action create (up to date)
* ruby_block[wait for patroni service socket] action run (skipped due to not_if)
* directory[/opt/gitlab/service/patroni/supervise] action create (up to date)
* directory[/opt/gitlab/service/patroni/log/supervise] action create (up to date)
* file[/opt/gitlab/service/patroni/supervise/ok] action touch (skipped due to only_if)
* file[/opt/gitlab/service/patroni/log/supervise/ok] action touch (skipped due to only_if)
* file[/opt/gitlab/service/patroni/supervise/status] action touch (skipped due to only_if)
* file[/opt/gitlab/service/patroni/log/supervise/status] action touch
- change owner from 'root' to 'gitlab-psql'
- change group from 'root' to 'gitlab-psql'
- update utime on file /opt/gitlab/service/patroni/log/supervise/status
* file[/opt/gitlab/service/patroni/supervise/control] action touch (skipped due to only_if)
* file[/opt/gitlab/service/patroni/log/supervise/control] action touch (skipped due to only_if)
* ruby_block[wait for node bootstrap to complete] action nothing (skipped due to action :nothing)
* execute[update dynamic configuration settings] action nothing (skipped due to action :nothing)
* execute[update dynamic configuration settings] action nothing (skipped due to action :nothing)
* ruby_block[wait for postgresql to start] action run
- execute the ruby block wait for postgresql to start
* execute[reload postgresql] action nothing (skipped due to action :nothing)
* database_objects[patroni] action create (skipped due to not_if)
* execute[signal to restart postgresql] action run
- Would execute /opt/gitlab/embedded/bin/patronictl -c /var/opt/gitlab/patroni/patroni.yaml restart --force postgresql-ha dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal
* ruby_block[wait for node bootstrap to complete] action run
- execute the ruby block wait for node bootstrap to complete
* execute[signal to restart postgresql] action run
[execute] + Cluster: postgresql-ha (uninitialized) -------------------------+-------------+---------+---------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-----------------------------------------------------------------+-------------+---------+---------+----+-----------+
| dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal | 10.164.0.20 | Replica | running | 2 | 0 |
+-----------------------------------------------------------------+-------------+---------+---------+----+-----------+
Traceback (most recent call last):
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1369, in getresponse
response.begin()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 310, in begin
version, status, reason = self._read_status()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 271, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/gitlab/embedded/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/gitlab/embedded/bin/patronictl", line 8, in <module>
sys.exit(ctl())
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/ctl.py", line 584, in restart
r = request_patroni(member, 'post', 'restart', content)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/ctl.py", line 169, in request_patroni
return request_executor(member, method, endpoint, data)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/request.py", line 53, in __call__
return self.request(method, url, data, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/request.py", line 46, in request
return self._pool.request(method.upper(), url, body=body, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/request.py", line 79, in request
method, url, fields=fields, headers=headers, **urlopen_kw
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/request.py", line 170, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/poolmanager.py", line 375, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/util/retry.py", line 531, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1369, in getresponse
response.begin()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 310, in begin
version, status, reason = self._read_status()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 271, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/gitlab/embedded/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
================================================================================
Error executing action `run` on resource 'execute[signal to restart postgresql]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/patronictl -c /var/opt/gitlab/patroni/patroni.yaml restart --force postgresql-ha dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal ----
STDOUT: + Cluster: postgresql-ha (uninitialized) -------------------------+-------------+---------+---------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-----------------------------------------------------------------+-------------+---------+---------+----+-----------+
| dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal | 10.164.0.20 | Replica | running | 2 | 0 |
+-----------------------------------------------------------------+-------------+---------+---------+----+-----------+
STDERR: Traceback (most recent call last):
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1369, in getresponse
response.begin()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 310, in begin
version, status, reason = self._read_status()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 271, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/gitlab/embedded/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/gitlab/embedded/bin/patronictl", line 8, in <module>
sys.exit(ctl())
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/ctl.py", line 584, in restart
r = request_patroni(member, 'post', 'restart', content)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/ctl.py", line 169, in request_patroni
return request_executor(member, method, endpoint, data)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/request.py", line 53, in __call__
return self.request(method, url, data, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/request.py", line 46, in request
return self._pool.request(method.upper(), url, body=body, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/request.py", line 79, in request
method, url, fields=fields, headers=headers, **urlopen_kw
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/request.py", line 170, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/poolmanager.py", line 375, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/util/retry.py", line 531, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1369, in getresponse
response.begin()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 310, in begin
version, status, reason = self._read_status()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 271, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/gitlab/embedded/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
---- End output of /opt/gitlab/embedded/bin/patronictl -c /var/opt/gitlab/patroni/patroni.yaml restart --force postgresql-ha dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal ----
Ran /opt/gitlab/embedded/bin/patronictl -c /var/opt/gitlab/patroni/patroni.yaml restart --force postgresql-ha dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal returned 1
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb
121: execute 'signal to restart postgresql' do
122: command "#{patroni_helper.ctl_command} -c #{patroni_config_file} restart --force #{node['patroni']['scope']} #{node['patroni']['name']}"
123: not_if { patroni_helper.repmgr_data_present? }
124: only_if { omnibus_helper.service_dir_enabled?('postgresql') && patroni_helper.node_status == 'running' }
125: notifies :run, 'ruby_block[wait for node bootstrap to complete]', :before
126: end
127:
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/patroni/recipes/enable.rb:121:in `from_file'
execute("signal to restart postgresql") do
action [:run]
updated true
default_guard_interpreter :execute
command "/opt/gitlab/embedded/bin/patronictl -c /var/opt/gitlab/patroni/patroni.yaml restart --force postgresql-ha dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal"
backup 5
declared_type :execute
cookbook_name "patroni"
recipe_name "enable"
domain nil
user nil
not_if { #code block }
only_if { #code block }
end
System Info:
------------
chef_version=15.14.0
platform=ubuntu
platform_version=20.04
ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Recipe: consul::enable_daemon
* execute[reload consul] action run
- execute /opt/gitlab/bin/gitlab-ctl hup consul
Recipe: patroni::enable
* execute[update dynamic configuration settings] action run (skipped due to only_if)
* execute[update dynamic configuration settings] action run (skipped due to only_if)
Running handlers:
There was an error running gitlab-ctl reconfigure:
execute[signal to restart postgresql] (patroni::enable line 121) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/patronictl -c /var/opt/gitlab/patroni/patroni.yaml restart --force postgresql-ha dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal ----
STDOUT: + Cluster: postgresql-ha (uninitialized) -------------------------+-------------+---------+---------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-----------------------------------------------------------------+-------------+---------+---------+----+-----------+
| dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal | 10.164.0.20 | Replica | running | 2 | 0 |
+-----------------------------------------------------------------+-------------+---------+---------+----+-----------+
STDERR: Traceback (most recent call last):
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1369, in getresponse
response.begin()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 310, in begin
version, status, reason = self._read_status()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 271, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/gitlab/embedded/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/gitlab/embedded/bin/patronictl", line 8, in <module>
sys.exit(ctl())
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/ctl.py", line 584, in restart
r = request_patroni(member, 'post', 'restart', content)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/ctl.py", line 169, in request_patroni
return request_executor(member, method, endpoint, data)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/request.py", line 53, in __call__
return self.request(method, url, data, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/request.py", line 46, in request
return self._pool.request(method.upper(), url, body=body, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/request.py", line 79, in request
method, url, fields=fields, headers=headers, **urlopen_kw
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/request.py", line 170, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/poolmanager.py", line 375, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/util/retry.py", line 531, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1369, in getresponse
response.begin()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 310, in begin
version, status, reason = self._read_status()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 271, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/gitlab/embedded/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
---- End output of /opt/gitlab/embedded/bin/patronictl -c /var/opt/gitlab/patroni/patroni.yaml restart --force postgresql-ha dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal ----
Ran /opt/gitlab/embedded/bin/patronictl -c /var/opt/gitlab/patroni/patroni.yaml restart --force postgresql-ha dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal returned 1
Running handlers complete
Chef Infra Client failed. 48 resources updated in 23 seconds$ sudo gitlab-ctl tail patroni
2021-01-26_18:31:54.85916 2021-01-26 18:31:54,859 INFO: No PostgreSQL configuration items changed, nothing to reload.
2021-01-26_18:31:54.86324 2021-01-26 18:31:54,863 INFO: establishing a new patroni connection to the postgres cluster
2021-01-26_18:31:54.87563 2021-01-26 18:31:54,874 ERROR: No initialize key in DCS and PostgreSQL is running as replica, aborting start
2021-01-26_18:31:54.87572 2021-01-26 18:31:54,874 ERROR: Please first start Patroni on the node running as master
2021-01-26_18:31:54.89290 2021-01-26 18:31:54,892 INFO: Register service postgresql-ha, params {'service_id': 'postgresql-ha/dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal', 'address': '10.164.0.20', 'port': 5432, 'check': {'http': 'http://10.164.0.20:8008/replica', 'interval': '10s', 'DeregisterCriticalServiceAfter': '150.0s'}, 'tags': ['replica']}$ sudo /opt/gitlab/embedded/bin/patronictl -c /var/opt/gitlab/patroni/patroni.yaml restart --force postgresql-ha dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal
+ Cluster: postgresql-ha (uninitialized) -------------------------+-------------+---------+---------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-----------------------------------------------------------------+-------------+---------+---------+----+-----------+
| dbalexandre-patroni-1-secondary-geo.c.group-geo-f9c951.internal | 10.164.0.20 | Replica | running | 2 | 0 |
+-----------------------------------------------------------------+-------------+---------+---------+----+-----------+
Traceback (most recent call last):
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1369, in getresponse
response.begin()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 310, in begin
version, status, reason = self._read_status()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 271, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/gitlab/embedded/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/gitlab/embedded/bin/patronictl", line 8, in <module>
sys.exit(ctl())
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/ctl.py", line 584, in restart
r = request_patroni(member, 'post', 'restart', content)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/ctl.py", line 169, in request_patroni
return request_executor(member, method, endpoint, data)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/request.py", line 53, in __call__
return self.request(method, url, data, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/patroni/request.py", line 46, in request
return self._pool.request(method.upper(), url, body=body, **kwargs)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/request.py", line 79, in request
method, url, fields=fields, headers=headers, **urlopen_kw
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/request.py", line 170, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/poolmanager.py", line 375, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/util/retry.py", line 531, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/opt/gitlab/embedded/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 1369, in getresponse
response.begin()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 310, in begin
version, status, reason = self._read_status()
File "/opt/gitlab/embedded/lib/python3.7/http/client.py", line 271, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/gitlab/embedded/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))$ sudo gitlab-ctl status
run: alertmanager: (pid 35884) 1535s; run: log: (pid 31215) 2387s
run: consul: (pid 35910) 1530s; run: log: (pid 32625) 2205s
run: geo-logcursor: (pid 35924) 1528s; run: log: (pid 32565) 2212s
run: geo-postgresql: (pid 35928) 1528s; run: log: (pid 32435) 2220s
run: gitaly: (pid 35937) 1528s; run: log: (pid 30618) 2516s
run: gitlab-exporter: (pid 35956) 1527s; run: log: (pid 31146) 2407s
run: gitlab-workhorse: (pid 35958) 1527s; run: log: (pid 31036) 2432s
run: grafana: (pid 35966) 1527s; run: log: (pid 31516) 2306s
run: logrotate: (pid 35979) 1526s; run: log: (pid 31094) 2417s
run: nginx: (pid 35986) 1526s; run: log: (pid 31068) 2425s
run: node-exporter: (pid 35992) 1525s; run: log: (pid 31135) 2411s
down: patroni: 1s, normally up, want up; run: log: (pid 38549) 733s
run: postgres-exporter: (pid 35999) 1525s; run: log: (pid 31232) 2383s
run: postgresql: (pid 36008) 1524s; run: log: (pid 30772) 2508s
run: prometheus: (pid 36025) 1524s; run: log: (pid 31187) 2395s
run: puma: (pid 36036) 1523s; run: log: (pid 30979) 2444s
run: redis: (pid 36041) 1523s; run: log: (pid 30576) 2522s
run: redis-exporter: (pid 36046) 1523s; run: log: (pid 31168) 2399s
run: sidekiq: (pid 36057) 1518s; run: log: (pid 31000) 2438sEdited by Douglas Barbosa Alexandre