cassandra client going stale when scylla goes down
WARNING:cassandra.cluster:Host 10.42.1.48:9042 has been marked down
WARNING:cassandra.cluster:[control connection] Error connecting to 10.42.1.48:9042:
Traceback (most recent call last):
File "cassandra/cluster.py", line 2928, in cassandra.cluster.ControlConnection._reconnect_internal
File "cassandra/cluster.py", line 2950, in cassandra.cluster.ControlConnection._try_connect
File "cassandra/cluster.py", line 1292, in cassandra.cluster.Cluster.connection_factory
File "cassandra/connection.py", line 471, in cassandra.connection.Connection.factory
File "/usr/local/lib/python3.7/dist-packages/cassandra/io/asyncorereactor.py", line 341, in __init__
self._connect_socket()
File "cassandra/connection.py", line 514, in cassandra.connection.Connection._connect_socket
ConnectionRefusedError: [Errno 111] Tried connecting to [('10.42.1.48', 9042)]. Last error: Connection refused
WARNING:cassandra.connection:Heartbeat failed for connection (140329468799576) to 10.42.1.48:9042
WARNING:cassandra.connection:Heartbeat failed for connection (140329468594216) to 10.42.1.48:9042
WARNING:cassandra.cluster:Host 10.42.1.48:9042 has been marked down
WARNING:cassandra.connection:Heartbeat failed for connection (140329468799576) to 10.42.1.48:9042
WARNING:cassandra.connection:Heartbeat failed for connection (140329468594216) to 10.42.1.48:9042
WARNING:cassandra.cluster:Host 10.42.1.48:9042 has been marked down
Traceback (most recent call last):
File "falcon/api.py", line 274, in falcon.api.API.__call__
File "falcon/api.py", line 269, in falcon.api.API.__call__
File "./icarus/api/__init__.py", line 29, in __fn
responder(self, req, resp, *args, **kwargs)
File "./icarus/api/__init__.py", line 67, in on_get
for state in self.db.get_states(timestamp, bbox)
File "./icarus/db/__init__.py", line 241, in get_states
s2cells,
File "cassandra/cluster.py", line 2240, in cassandra.cluster.Session.execute
File "cassandra/cluster.py", line 4198, in cassandra.cluster.ResponseFuture.result
cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {})
It keeps saying that even though scylla ATM is fine.