fix(lib): TCP/IP Client becomes permanently broken after any recv() timeout

Summary

Once any remote call on a _DeviceTCPIPClient times out, every subsequent call on that same client fails — even if the server is healthy and the hardware has recovered. The client must be rebuilt from scratch to recover.

Root cause

_send() in tcp_ip_client.py sets RCVTIMEO on the socket, calls send(), then recv(). If recv() raises zmq.error.Again (timeout expired), it is caught and re-raised as TimeoutError:

try: recv_timeout = self._resolve_recv_timeout_ms(payload.get("timeout", None)) self.socket.setsockopt(zmq.RCVTIMEO, recv_timeout) self.socket.send(json.dumps(payload).encode()) reply = self.socket.recv() # ← raises Again on timeout except zmq.error.Again: raise TimeoutError("Server did not respond in time")

The problem: the ZMQ REQ socket follows a strict send→recv→send state machine. After send() succeeds, the socket transitions to RECV state. If recv() times out without consuming a reply, the socket stays in RECV state. There is no cleanup in the except block.

The next call to _send() reaches socket.send() while the socket is in RECV state, which raises zmq.error.ZMQError: Operation cannot be accomplished in current state (EFSM) — or another Again depending on ZMQ version — making the client permanently unusable until the process reconnects.

Observed symptom

In a function that makes multiple sequential AMC calls:

def compute_stage_step(self): pre_pos = self.amc.get_position(0)["position"] # sometimes TimeoutError self.amc.set_n_steps(0, 1, True) # sometimes TimeoutError post_pos = self.amc.get_position(0)["position"]

The failure rotates between calls across invocations:

  • If set_n_steps times out in run N, get_position fails in run N+1 (socket stuck from the previous timeout).
  • If get_position times out in run N, get_position fails again in run N+1.

The same server and hardware work reliably when called from localhost — only remote clients with real network latency trigger the timeout threshold.

Suggested fixes

Option 1 — Reconnect after timeout (minimal change)

Reset the socket in the except block so the client can be reused:

except zmq.error.Again: # REQ socket is stuck in RECV state; reconnect to restore SEND state self.socket.close() self.socket = self._context.socket(zmq.REQ) self.socket.setsockopt(zmq.RCVTIMEO, self.default_timeout) self.socket.connect(self._address) raise TimeoutError("Server did not respond in time")

Option 2 — Switch to DEALER socket (robust, more involved)

Replace REQ with DEALER on the client side. DEALER does not enforce a send→recv order, so a missed reply does not corrupt socket state. Requires adding a message envelope (empty delimiter frame) to match the REP server's framing.

Option 3 — Per-call timeout via server payload (avoid the problem)

For long-running hardware operations, pass the expected duration as the server-side timeout field in the payload so the server returns a timeout/error reply itself (within the grace window) rather than silently hanging. This keeps the ZMQ exchange intact and avoids ever triggering RCVTIMEO. Already partially supported by _resolve_recv_timeout_ms.

Option 4 — Non-blocking server functions (architectural)

Server-side set_n_steps and similar fire-and-forget hardware commands should return immediately after dispatching the serial command, rather than blocking until hardware ACK. The client then polls position or sleeps. This eliminates the long-duration blocking calls that make timeouts likely over real networks.

Environment

  • plesty-lib 0.3.0.dev1+2d3f904c
  • ZMQ transport: tcp:// (remote hardware, not localhost)
  • Python 3.13