potential split brain during patroni failover in staging
We had a patroni failover event on 2019/02/24 01:26:55. potential split brain may happened during this failover:
2019/02/24 01:26:50
04
node promoted to new master
patroni log:
2019-02-24_01:26:50 patroni-04-db-gstg patroni[19496]: 2019-02-24 01:26:50,789 INFO: Got response from patroni-02-db-gstg.c.gitlab-staging-1.internal http://10.224.29.102:8009/patroni: b'{"timeline": 69, "database_system_identifier": "6343687859876602183", "role": "replica", "xlog": {"received_location": 60090049123248, "paused": false, "replayed_location": 60090049123248, "replayed_timestamp": "2019-02-24 01:26:38.970 GMT"}, "server_version": 90611, "patroni": {"version": "1.5.0", "scope": "pg-ha-cluster"}, "cluster_unlocked": true, "state": "running", "postmaster_start_time": "2019-02-21 22:36:24.855 GMT"}'
2019-02-24_01:26:51 patroni-04-db-gstg patroni[19496]: 2019-02-24 01:26:51,015 INFO: Got response from patroni-01-db-gstg.c.gitlab-staging-1.internal http://10.224.29.101:8009/patroni: b'{"xlog": {"replayed_location": 60090049123248, "received_location": 60090049123248, "replayed_timestamp": "2019-02-24 01:26:38.970 GMT", "paused": false}, "server_version": 90611, "state": "running", "cluster_unlocked": true, "database_system_identifier": "6343687859876602183", "role": "replica", "timeline": 69, "postmaster_start_time": "2019-02-19 01:05:45.977 GMT", "patroni": {"scope": "pg-ha-cluster", "version": "1.5.0"}}'
2019-02-24_01:26:51 patroni-04-db-gstg patroni[19496]: 2019-02-24 01:26:51,026 INFO: Got response from patroni-03-db-gstg.c.gitlab-staging-1.internal http://10.224.29.103:8009/patroni: b'{"timeline": 69, "role": "replica", "postmaster_start_time": "2019-02-19 01:05:45.903 GMT", "patroni": {"scope": "pg-ha-cluster", "version": "1.5.0"}, "xlog": {"replayed_timestamp": "2019-02-24 01:26:38.970 GMT", "replayed_location": 60090049123248, "received_location": 60090049123248, "paused": false}, "server_version": 90611, "state": "running", "database_system_identifier": "6343687859876602183", "cluster_unlocked": true}'
2019-02-24_01:26:51 patroni-04-db-gstg patroni[19496]: 2019-02-24 01:26:51,139 INFO: Got response from patroni-05-db-gstg.c.gitlab-staging-1.internal http://10.224.29.105:8009/patroni: b'{"database_system_identifier": "6343687859876602183", "patroni": {"version": "1.5.0", "scope": "pg-ha-cluster"}, "postmaster_start_time": "2019-02-19 01:05:45.718 GMT", "state": "running", "role": "replica", "xlog": {"replayed_location": 60090049123248, "paused": false, "received_location": 60090049123248, "replayed_timestamp": "2019-02-24 01:26:38.970 GMT"}, "server_version": 90611, "cluster_unlocked": true, "timeline": 69}'
2019-02-24_01:26:52 patroni-04-db-gstg patroni[19496]: 2019-02-24 01:26:52,765 WARNING: Request failed to patroni-06-db-gstg.c.gitlab-staging-1.internal: GET http://10.224.29.106:8009/patroni (HTTPConnectionPool(host='10.224.29.106', port=8009): Read timed out. (read timeout=2))
2019-02-24_01:26:52 patroni-04-db-gstg patroni[19496]: 2019-02-24 01:26:52,873 WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"
2019-02-24_01:26:52 patroni-04-db-gstg patroni[19496]: 2019-02-24 01:26:52,874 INFO: promoted self to leader by acquiring session lock
2019-02-24_01:26:52 patroni-04-db-gstg patroni[19496]: server promoting
2019-02-24_01:26:52 patroni-04-db-gstg patroni[19496]: 2019-02-24 01:26:52,906 INFO: cleared rewind state after becoming the leader
2019-02-24_01:26:53 patroni-04-db-gstg patroni[19496]: 2019-02-24 01:26:53,947 INFO: Lock owner: patroni-04-db-gstg.c.gitlab-staging-1.internal; I am patroni-04-db-gstg.c.gitlab-staging-1.internal
postgres log
2019-02-24_01:26:52 patroni-04-db-gstg postgres[3107]: [7-1] 2019-02-24 01:26:52 GMT [3107]: [6-1] LOG: received promote request
2019-02-24_01:26:52 patroni-04-db-gstg postgres[3124]: [4-1] 2019-02-24 01:26:52 GMT [3124]: [2-1] FATAL: terminating walreceiver process due to administrator command
2019-02-24_01:26:52 patroni-04-db-gstg postgres[3107]: [8-1] 2019-02-24 01:26:52 GMT [3107]: [7-1] LOG: record with incorrect prev-link 1716AB00/0 at 36A6/CE0853E8
2019-02-24_01:26:52 patroni-04-db-gstg postgres[3107]: [9-1] 2019-02-24 01:26:52 GMT [3107]: [8-1] LOG: redo done at 36A6/CE0853B0
2019-02-24_01:26:52 patroni-04-db-gstg postgres[3107]: [10-1] 2019-02-24 01:26:52 GMT [3107]: [9-1] LOG: last completed transaction was at log time 2019-02-24 01:26:38.970079+00
2019-02-24_01:26:52 patroni-04-db-gstg postgres[3107]: [11-1] 2019-02-24 01:26:52 GMT [3107]: [10-1] LOG: selected new timeline ID: 70
2019-02-24_01:26:52 patroni-04-db-gstg postgres[3107]: [12-1] 2019-02-24 01:26:52 GMT [3107]: [11-1] LOG: archive recovery complete
2019-02-24_01:26:52 patroni-04-db-gstg postgres[3107]: [13-1] 2019-02-24 01:26:52 GMT [3107]: [12-1] LOG: MultiXact member wraparound protections are now enabled
2019-02-24_01:26:53 patroni-04-db-gstg postgres[14479]: [3-1] 2019-02-24 01:26:53 GMT [14479]: [1-1] LOG: autovacuum launcher started
2019-02-24_01:26:53 patroni-04-db-gstg postgres[3105]: [3-1] 2019-02-24 01:26:53 GMT [3105]: [4-1] LOG: database system is ready to accept connections
2019/02/24 01:26:56
06
node (Old master) patroni started failing
patroni log
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: 2019-02-24 01:26:42,277 ERROR: get_cluster
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: Traceback (most recent call last):
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/urllib3/connectionpool.py", line 384, in _make_request
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: six.raise_from(e, None)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "<string>", line 2, in raise_from
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/urllib3/connectionpool.py", line 380, in _make_request
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: httplib_response = conn.getresponse()
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: response.begin()
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/http/client.py", line 297, in begin
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: version, status, reason = self._read_status()
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/socket.py", line 575, in readinto
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: return self._sock.recv_into(b)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: socket.timeout: timed out
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: During handling of the above exception, another exception occurred:
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: Traceback (most recent call last):
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: chunked=chunked)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/urllib3/connectionpool.py", line 386, in _make_request
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=8500): Read timed out. (read timeout=3.3333333333333335)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: During handling of the above exception, another exception occurred:
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: Traceback (most recent call last):
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/patroni/utils.py", line 254, in __call__
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: return func(*args, **kwargs)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/consul/base.py", line 554, in get
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: params=params)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/patroni/dcs/consul.py", line 105, in wrapper
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: return callback(self.response(self.http.request(method.upper(), self.uri(path, params), **kwargs)))
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/urllib3/request.py", line 68, in request
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: **urlopen_kw)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/urllib3/request.py", line 89, in request_encode_url
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: return self.urlopen(method, url, **extra_kw)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/urllib3/poolmanager.py", line 323, in urlopen
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: response = conn.urlopen(method, u.request_uri, **kw)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/urllib3/connectionpool.py", line 638, in urlopen
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: _stacktrace=sys.exc_info()[2])
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/urllib3/util/retry.py", line 398, in increment
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: raise MaxRetryError(_pool, url, error or ResponseError(cause))
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/service/pg-ha-cluster/?recurse=1 (Caused by ReadTimeoutError("HTTPConnectionPool(host='127.0.0.1', port=8500): Read timed out. (read timeout=3.3333333333333335)",))
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: During handling of the above exception, another exception occurred:
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: Traceback (most recent call last):
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/patroni/dcs/consul.py", line 283, in _load_cluster
019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: _, results = self.retry(self._client.kv.get, path, recurse=True)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/patroni/dcs/consul.py", line 214, in retry
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: return self._retry.copy()(*args, **kwargs)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/patroni/utils.py", line 263, in __call__
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: raise RetryFailedError("Exceeded retry deadline")
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: patroni.utils.RetryFailedError: 'Exceeded retry deadline'
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: 2019-02-24 01:26:56,599 ERROR: Error communicating with DCS
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: Traceback (most recent call last):
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/socketserver.py", line 625, in process_request_thread
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self.finish_request(request, client_address)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/socketserver.py", line 354, in finish_request
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self.RequestHandlerClass(request, client_address, self)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/socketserver.py", line 681, in __init__
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self.handle()
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/http/server.py", line 422, in handle
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self.handle_one_request()
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/http/server.py", line 410, in handle_one_request
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: method()
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/patroni/api.py", line 108, in do_GET
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self._write_status_response(status_code, response)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/patroni/api.py", line 76, in _write_status_response
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self._write_json_response(status_code, response)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/patroni/api.py", line 47, in _write_json_response
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self._write_response(status_code, json.dumps(response), content_type='application/json')
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/patroni/api.py", line 44, in _write_response
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self.wfile.write(body.encode('utf-8'))
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/socket.py", line 593, in write
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: return self._sock.send(b)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: BrokenPipeError: [Errno 32] Broken pipe
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: Traceback (most recent call last):
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/socketserver.py", line 625, in process_request_thread
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self.finish_request(request, client_address)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/socketserver.py", line 354, in finish_request
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self.RequestHandlerClass(request, client_address, self)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/socketserver.py", line 681, in __init__
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self.handle()
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/http/server.py", line 422, in handle
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self.handle_one_request()
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/usr/lib/python3.5/http/server.py", line 410, in handle_one_request
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: method()
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/patroni/api.py", line 108, in do_GET
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self._write_status_response(status_code, response)
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: File "/opt/patroni/lib/python3.5/site-packages/patroni/api.py", line 76, in _write_status_response
2019-02-24_01:26:56 patroni-06-db-gstg patroni[318]: self._write_json_response(status_code, response)
...
2019-02-24 01:27:16
06
node postgres received shutdown
postgres log
2019-02-24_01:27:16 patroni-06-db-gstg postgres[5005]: [6-1] 2019-02-24 01:27:16 GMT [5005]: [7-1] LOG: received fast shutdown request
2019-02-24_01:27:18 patroni-06-db-gstg postgres[5005]: [7-1] 2019-02-24 01:27:18 GMT [5005]: [8-1] LOG: aborting any active transactions
2019-02-24_01:27:18 patroni-06-db-gstg postgres[5622]: [11-1] 2019-02-24 01:27:18 GMT [5622]: [6-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[3269]: [18-1] 2019-02-24 01:27:18 GMT [3269]: [13-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[829]: [34-1] 2019-02-24 01:27:18 GMT [829]: [29-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[829]: [34-2] 2019-02-24 01:27:18 GMT [829]: [30-1] STATEMENT: SELECT COUNT(*) FROM "projects" INNER JOIN "project_repository_states" ON "project_repository_states"."project_id" = "projects"."id" WHERE ("project_repository_states"."wiki_verification_checksum" IS NOT NULL) AND "project_repository_states"."last_wiki_verification_failure" IS NULL
2019-02-24_01:27:18 patroni-06-db-gstg postgres[648]: [25-1] 2019-02-24 01:27:18 GMT [648]: [20-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[2068]: [16-1] 2019-02-24 01:27:18 GMT [2068]: [11-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[29546]: [56-1] 2019-02-24 01:27:18 GMT [29546]: [51-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[2677]: [14-1] 2019-02-24 01:27:18 GMT [2677]: [9-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[30495]: [42-1] 2019-02-24 01:27:18 GMT [30495]: [37-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[2626]: [6-1] 2019-02-24 01:27:18 GMT [2626]: [1-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[29548]: [53-1] 2019-02-24 01:27:18 GMT [29548]: [48-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[32536]: [28-1] 2019-02-24 01:27:18 GMT [32536]: [23-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[32534]: [34-1] 2019-02-24 01:27:18 GMT [32534]: [29-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[32000]: [35-1] 2019-02-24 01:27:18 GMT [32000]: [30-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[3567]: [6-1] 2019-02-24 01:27:18 GMT [3567]: [1-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[30489]: [50-1] 2019-02-24 01:27:18 GMT [30489]: [45-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[29549]: [50-1] 2019-02-24 01:27:18 GMT [29549]: [45-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[29544]: [54-1] 2019-02-24 01:27:18 GMT [29544]: [49-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[2067]: [21-1] 2019-02-24 01:27:18 GMT [2067]: [16-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[28262]: [61-1] 2019-02-24 01:27:18 GMT [28262]: [56-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[5443]: [16-1] 2019-02-24 01:27:18 GMT [5443]: [11-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[26141]: [76-1] 2019-02-24 01:27:18 GMT [26141]: [71-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[25566]: [83-1] 2019-02-24 01:27:18 GMT [25566]: [78-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[22890]: [91-1] 2019-02-24 01:27:18 GMT [22890]: [86-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[7462]: [6-1] 2019-02-24 01:27:18 GMT [7462]: [1-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[5133]: [4-1] 2019-02-24 01:27:18 GMT [5133]: [2-1] LOG: autovacuum launcher shutting down
2019-02-24_01:27:18 patroni-06-db-gstg postgres[330]: [5-1] 2019-02-24 01:27:18 GMT [330]: [1-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[30142]: [48-1] 2019-02-24 01:27:18 GMT [30142]: [43-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[1200]: [27-1] 2019-02-24 01:27:18 GMT [1200]: [22-1] FATAL: terminating connection due to administrator command
2019-02-24_01:27:18 patroni-06-db-gstg postgres[6064]: [8-1] 2019-02-24 01:27:18 GMT [6064]: [1-1] FATAL: the database system is shutting down
2019-02-24_01:27:19 patroni-06-db-gstg postgres[5014]: [14009-1] 2019-02-24 01:27:19 GMT [5014]: [14008-1] LOG: checkpoint complete: wrote 6924 buffers (0.7%); 0 transaction log file(s) added, 0 removed, 4 recycled; write=103.902 s, sync=0.016 s, total=103.971 s; sync files=435, longest=0.002 s, average=0.000 s; distance=96600 kB, estimate=96600 kB
2019-02-24_01:27:19 patroni-06-db-gstg postgres[5014]: [14010-1] 2019-02-24 01:27:19 GMT [5014]: [14009-1] LOG: shutting down
2019-02-24_01:27:19 patroni-06-db-gstg postgres[5014]: [14011-1] 2019-02-24 01:27:19 GMT [5014]: [14010-1] LOG: checkpoint starting: shutdown immediate
2019-02-24_01:27:19 patroni-06-db-gstg postgres[5014]: [14012-1] 2019-02-24 01:27:19 GMT [5014]: [14011-1] LOG: checkpoint complete: wrote 1278 buffers (0.1%); 0 transaction log file(s) added, 0 removed, 1 recycled; write=0.057 s, sync=0.002 s, total=0.094 s; sync files=27, longest=0.001 s, average=0.000 s; distance=48551 kB, estimate=91795 kB
2019-02-24_01:27:21 patroni-06-db-gstg postgres[5005]: [8-1] 2019-02-24 01:27:21 GMT [5005]: [9-1] LOG: online backup mode canceled
2019-02-24_01:27:21 patroni-06-db-gstg postgres[5005]: [8-2] 2019-02-24 01:27:21 GMT [5005]: [10-1] DETAIL: File "backup_label" was renamed to "backup_label.old".
2019-02-24_01:27:23 patroni-06-db-gstg postgres[5005]: [9-1] 2019-02-24 01:27:23 GMT [5005]: [11-1] LOG: database system is shut down
between 01:26:56 - 01:27:16 the old master node logged some select queries, so probably the connection were still served when the new master was promoted
Edited by Andreas Brandl