fix(ipython): prevent kernel shutdown deadlock from hanging the service

What happened

A shared service kernel stopped answering tasks. The task manager logged one warning each interval:

2026-08-28 10:08:15,489 - orangeqs.juice.task_manager._kernel_monitor - WARNING - device: Kernel is not ready yet. Kernel didn't respond to heartbeats in 1 seconds and timed out

No other errors appeared in the task manager logs or the device service logs.

The device service process was still alive. Supervisor reported it as RUNNING with an uptime of almost 4 days:

juice-service   RUNNING   pid 134, uptime 3 days, 23:53:57

But the kernel served nothing. A direct client probe failed at the Jupyter handshake:

RuntimeError: Kernel died before replying to kernel_info

Root cause

The kernel process was stuck inside shutdown. A py-spy dump of pid 134 shows the main thread blocked forever in IPKernelApp.close():

Process 134: /env/.venv/bin/python3 /env/.venv/bin/juice service device
Python v3.12.12 (/usr/bin/python3.12)

Thread 134 (idle): "MainThread"
    _wait_for_tstate_lock (threading.py:1169)
    join (threading.py:1149)
    close (ipykernel/kernelapp.py:408)
Thread 191 (idle): "Thread-1 (_watch_pipe_fd)"
    _watch_pipe_fd (ipykernel/iostream.py:387)
    run (threading.py:1012)
    _bootstrap_inner (threading.py:1075)
    _bootstrap (threading.py:1032)
Thread 192 (idle): "Thread-2 (_watch_pipe_fd)"
    _watch_pipe_fd (ipykernel/iostream.py:387)
    run (threading.py:1012)
    _bootstrap_inner (threading.py:1075)
    _bootstrap (threading.py:1032)
Thread 193 (idle): "Control"
    send (zmq/sugar/socket.py:713)
    schedule (ipykernel/iostream.py:267)
    send_multipart (ipykernel/iostream.py:276)
    send_multipart (ipykernel/iostream.py:346)
    send (jupyter_client/session.py:867)
    _publish_status (ipykernel/kernelbase.py:607)
    _publish_status_and_flush (ipykernel/kernelbase.py:617)
    process_control (ipykernel/kernelbase.py:339)
    dispatch_control (ipykernel/kernelbase.py:302)
    start (tornado/platform/asyncio.py:211)
    run (ipykernel/control.py:23)
    _bootstrap_inner (threading.py:1075)
    _bootstrap (threading.py:1032)
Thread 198 (idle): "Thread-4 (run)"
    run (asyncio/runners.py:118)
    run (asyncio/runners.py:195)
    run (threading.py:1012)
    run_closure (ipykernel/ipkernel.py:772)
    _bootstrap_inner (threading.py:1075)
    _bootstrap (threading.py:1032)

IPKernelApp.close() runs at shutdown through atexit. It calls self.control_thread.join() with no timeout (kernelapp.py:408). The Control thread was blocked in a zmq send. It published an iopub status message while iopub was torn down. The join never returned.

The process never exited, so Supervisor never respawned it. The kernel became a permanent zombie. It was alive to the operating system and dead to its ZMQ channels. This is an ipykernel shutdown deadlock. Juice pins ipykernel <7, so an upgrade does not fix it.

Fix

IPythonService already subclasses the kernel application as _NoArgvIPKernelApp. This change overrides close() on that subclass to arm a shutdown watchdog:

• The watchdog is a daemon threading.Timer. It calls os._exit(0) after 20 seconds.
• A normal shutdown finishes in under a second and cancels the watchdog. Behavior does not change.
• A deadlocked close() triggers the watchdog. The process exits and Supervisor starts a fresh kernel.

The override guards every shutdown path: atexit, an explicit close, a SIGTERM, and a client shutdown request. A permanent hang becomes a bounded delay of at most 20 seconds.

Tests

New tests in tests/juice/service/test_ipython.py:

• test_force_exit_watchdog_fires_after_timeout and test_force_exit_watchdog_cancel_prevents_exit check the watchdog primitive.
• test_noargv_kernelapp_close_forces_exit_on_deadlock reproduces the deadlock. It patches IPKernelApp.close to hang and asserts the watchdog forces the exit.
• test_noargv_kernelapp_close_cancels_watchdog_on_clean_close checks that a clean close cancels the watchdog.

Result: 4 passed. The 4 pre-existing service-fixture errors are unrelated. They fail the same way on a clean tree because of a Python 3.14 event-loop issue.