Crash when restarting tango loop in the same process
In PyTango we have to run our tests with --forked option due to attempts to restart tango loop in the same process result in crash.
E.g. if I do:
from tango.server import Device
from tango.test_context import DeviceTestContext
class MyDevice(Device):
pass
if __name__ == "__main__":
with DeviceTestContext(MyDevice, timeout=60000, debug=5) as dev_ctx:
pass
print("First device context finished, now the second one")
with DeviceTestContext(MyDevice, timeout=60000, debug=5) as dev_ctx2:
passand run it, then I get crash here:
with SIGSEGV (signal SIGSEGV: address not mapped to object (fault address: 0x0))
Here is the output of shutdown process and restart:
Ready to accept request
Ready to accept request
2025-10-27T11:05:24,902275+0100 DEBUG (device.cpp:2405) dserver/MyDevice/MyDevice DeviceImpl::info arrived
2025-10-27T11:05:24,902289+0100 DEBUG (device.cpp:2496) dserver/MyDevice/MyDevice Leaving DeviceImpl::info
2025-10-27T11:05:24,902297+0100 DEBUG (subdev_diag.cpp:104) dserver/MyDevice/MyDevice SubDevDiag::get_associated_device() entering ...
2025-10-27T11:05:24,902303+0100 DEBUG (subdev_diag.cpp:110) dserver/MyDevice/MyDevice SubDevDiag::get_associated_device() found :
2025-10-27T11:05:24,902309+0100 DEBUG (subdev_diag.cpp:128) dserver/MyDevice/MyDevice SubDevDiag::register_sub_device() dev_name = sub_dev_name = tango://127.0.0.1:33297/dserver/MyDevice/mydevice#dbase=no
2025-10-27T11:05:24,902356+0100 DEBUG (device.cpp:2510) dserver/MyDevice/MyDevice DeviceImpl::ping arrived
2025-10-27T11:05:24,902365+0100 DEBUG (device.cpp:2522) dserver/MyDevice/MyDevice Leaving DeviceImpl::ping
2025-10-27T11:05:24,909683+0100 DEBUG (device.cpp:2405) dserver/MyDevice/MyDevice DeviceImpl::info arrived
2025-10-27T11:05:24,909711+0100 DEBUG (device.cpp:2496) dserver/MyDevice/MyDevice Leaving DeviceImpl::info
2025-10-27T11:05:24,909719+0100 DEBUG (subdev_diag.cpp:104) dserver/MyDevice/MyDevice SubDevDiag::get_associated_device() entering ...
2025-10-27T11:05:24,909725+0100 DEBUG (subdev_diag.cpp:110) dserver/MyDevice/MyDevice SubDevDiag::get_associated_device() found :
2025-10-27T11:05:24,909731+0100 DEBUG (subdev_diag.cpp:128) dserver/MyDevice/MyDevice SubDevDiag::register_sub_device() dev_name = sub_dev_name = tango://127.0.0.1:33297/test/nodb/mydevice#dbase=no
2025-10-27T11:05:24,909777+0100 DEBUG (device.cpp:2510) dserver/MyDevice/MyDevice DeviceImpl::ping arrived
2025-10-27T11:05:24,909786+0100 DEBUG (device.cpp:2522) dserver/MyDevice/MyDevice Leaving DeviceImpl::ping
2025-10-27T11:05:24,909861+0100 DEBUG (device_4.cpp:437) dserver/MyDevice/MyDevice Device_4Impl::command_inout_4 arrived, source = 2, command = Kill
2025-10-27T11:05:24,909872+0100 DEBUG (device_2.cpp:126) dserver/MyDevice/MyDevice Device_2Impl::command_inout_2 arrived, source = 2, command = Kill
2025-10-27T11:05:24,909879+0100 DEBUG (tango_monitor.h:155) dserver/MyDevice/MyDevice In get_monitor() dserver/MyDevice/MyDevice cache, thread = 0, ctr = 0
2025-10-27T11:05:24,909910+0100 DEBUG (tango_monitor.h:206) dserver/MyDevice/MyDevice In rel_monitor() dserver/MyDevice/MyDevice cache, ctr = 1, thread = 0
2025-10-27T11:05:24,909916+0100 DEBUG (tango_monitor.h:216) dserver/MyDevice/MyDevice Signalling !
2025-10-27T11:05:24,909922+0100 DEBUG (tango_monitor.h:155) dserver/MyDevice/MyDevice In get_monitor() dserver/MyDevice/MyDevice, thread = 0, ctr = 0
2025-10-27T11:05:24,909928+0100 DEBUG (tango_monitor.h:155) dserver/MyDevice/MyDevice In get_monitor() dserver/MyDevice/MyDevice, thread = 0, ctr = 1
2025-10-27T11:05:24,909933+0100 DEBUG (tango_monitor.h:184) dserver/MyDevice/MyDevice owner_thread !!
2025-10-27T11:05:24,909938+0100 DEBUG (device.cpp:1795) dserver/MyDevice/MyDevice DeviceImpl::command_inout(): command received : Kill
2025-10-27T11:05:24,909943+0100 DEBUG (subdev_diag.cpp:104) dserver/MyDevice/MyDevice SubDevDiag::get_associated_device() entering ...
2025-10-27T11:05:24,909948+0100 DEBUG (subdev_diag.cpp:110) dserver/MyDevice/MyDevice SubDevDiag::get_associated_device() found :
2025-10-27T11:05:24,909953+0100 DEBUG (subdev_diag.cpp:83) dserver/MyDevice/MyDevice SubDevDiag::set_associated_device() entering ...
2025-10-27T11:05:24,909958+0100 DEBUG (deviceclass.cpp:1157) dserver/MyDevice/MyDevice Entering DeviceClass::command_handler() method
2025-10-27T11:05:24,909964+0100 DEBUG (dserverclass.cpp:417) dserver/MyDevice/MyDevice DevKillCmd::execute(): arrived
2025-10-27T11:05:24,909970+0100 DEBUG (dserver.cpp:1450) dserver/MyDevice/MyDevice In kill command
2025-10-27T11:05:24,910016+0100 DEBUG (deviceclass.cpp:1267) dserver/MyDevice/MyDevice Leaving DeviceClass::command_handler() method
2025-10-27T11:05:24,910023+0100 DEBUG (subdev_diag.cpp:83) dserver/MyDevice/MyDevice SubDevDiag::set_associated_device() entering ...
2025-10-27T11:05:24,910029+0100 DEBUG (device.cpp:1845) dserver/MyDevice/MyDevice DeviceImpl::command_inout(): leaving method for command Kill
2025-10-27T11:05:24,910034+0100 DEBUG (tango_monitor.h:206) dserver/MyDevice/MyDevice In rel_monitor() dserver/MyDevice/MyDevice, ctr = 2, thread = 0
2025-10-27T11:05:24,910039+0100 DEBUG (tango_monitor.h:206) dserver/MyDevice/MyDevice In rel_monitor() dserver/MyDevice/MyDevice, ctr = 1, thread = 0
2025-10-27T11:05:24,910044+0100 DEBUG (tango_monitor.h:216) dserver/MyDevice/MyDevice Signalling !
2025-10-27T11:05:24,910088+0100 DEBUG (dserver.cpp:1463) dserver/MyDevice/MyDevice In the killer thread !!!
2025-10-27T11:05:24,910149+0100 DEBUG (pollthread.cpp:752) dserver/MyDevice/MyDevice Received an exit command
2025-10-27T11:05:24,910208+0100 DEBUG (pollthread.cpp:239) dserver/MyDevice/MyDevice Received an unknown exception, probably about to exit.
2025-10-27T11:05:24,910282+0100 DEBUG (utils.cpp:2777) dserver/MyDevice/MyDevice Entering Util::unregister_server method
2025-10-27T11:05:24,910296+0100 DEBUG (utils.cpp:2800) dserver/MyDevice/MyDevice Leaving Util::unregister_server method
2025-10-27T11:05:24,910303+0100 DEBUG (deviceclass.cpp:870) dserver/MyDevice/MyDevice Entering DeviceClas::delete_dev method for device with index 0
2025-10-27T11:05:24,910355+0100 DEBUG (device.cpp:687) dserver/MyDevice/MyDevice Entering DeviceImpl destructor for device test/nodb/mydevice
2025-10-27T11:05:24,910374+0100 DEBUG (attribute.cpp:2507) dserver/MyDevice/MyDevice Attribute::delete_seq() called
2025-10-27T11:05:24,910380+0100 DEBUG (attribute.h:207) dserver/MyDevice/MyDevice AttrValue::reset()
2025-10-27T11:05:24,910387+0100 DEBUG (attribute.cpp:2507) dserver/MyDevice/MyDevice Attribute::delete_seq() called
2025-10-27T11:05:24,910392+0100 DEBUG (attribute.h:207) dserver/MyDevice/MyDevice AttrValue::reset()
2025-10-27T11:05:24,910403+0100 DEBUG (device.cpp:765) dserver/MyDevice/MyDevice Leaving DeviceImpl destructor for device test/nodb/mydevice
2025-10-27T11:05:24,910416+0100 DEBUG (deviceclass.cpp:904) dserver/MyDevice/MyDevice Leaving DeviceClass delete_dev
2025-10-27T11:05:24,910426+0100 DEBUG (tango_util.cpp:275) dserver/MyDevice/MyDevice PyUtil::cpp_device_class_delete called
2025-10-27T11:05:24,910436+0100 DEBUG (tango_util.cpp:282) dserver/MyDevice/MyDevice PyUtil::cpp_device_class_delete checking if MyDevice can be deleted in Cpp
2025-10-27T11:05:24,911014+0100 DEBUG (filedatabase.cpp:223) dserver/MyDevice/MyDevice FILEDATABASE: FileDatabase destructor
2025-10-27T11:05:24,911032+0100 DEBUG (utils_shut.cpp:132) dserver/MyDevice/MyDevice Database object deleted
2025-10-27T11:05:24,911039+0100 DEBUG (utils_shut.cpp:148) dserver/MyDevice/MyDevice Going to shutdown ORB
2025-10-27T11:05:24,911049+0100 DEBUG (device.cpp:687) dserver/MyDevice/MyDevice Entering DeviceImpl destructor for device dserver/MyDevice/MyDevice
2025-10-27T11:05:24,911059+0100 DEBUG (attribute.cpp:2507) dserver/MyDevice/MyDevice Attribute::delete_seq() called
2025-10-27T11:05:24,911065+0100 DEBUG (attribute.h:207) dserver/MyDevice/MyDevice AttrValue::reset()
2025-10-27T11:05:24,911071+0100 DEBUG (attribute.cpp:2507) dserver/MyDevice/MyDevice Attribute::delete_seq() called
2025-10-27T11:05:24,911076+0100 DEBUG (attribute.h:207) dserver/MyDevice/MyDevice AttrValue::reset()
2025-10-27T11:05:24,911086+0100 DEBUG (device.cpp:765) dserver/MyDevice/MyDevice Leaving DeviceImpl destructor for device dserver/MyDevice/MyDevice
2025-10-27T11:05:24,911314+0100 DEBUG (utils_shut.cpp:152) dserver/MyDevice/MyDevice ORB shutdown
2025-10-27T11:05:24,911449+0100 DEBUG (thsig.cpp:73) dserver/MyDevice/MyDevice ThSig stop requested by DSignalServer singleton
First device context finished, now the second one
Process finished with exit code 139 (interrupted by signal 11:SIGSEGV)Here is pystack analysis of core dump:
(tangodev_13) matveyev@haso102ym:~/pytango$ pystack core --native-all core
Using executable found in the core file: /home/matveyev/miniforge3/envs/tangodev_13/bin/python
Core file information:
state: D zombie: True niceness: 0
pid: 568902 ppid: 241453 sid: 241453
uid: 30593 gid: 39 pgrp: 568902
executable: python arguments: python sandbox/server_for_tests.py
The process died due receiving signal SIGSEGV
Traceback for thread 568918 [] (most recent call last):
(C) File "../sysdeps/unix/sysv/linux/x86_64/clone.S", line 95, in __clone (libc.so.6)
(C) File "./nptl/pthread_create.c", line 477, in start_thread (libpthread.so.0)
(C) File "/usr/local/src/conda/python-3.13.5/Python/thread_pthread.h", line 243, in pythread_wrapper (python)
(C) File "/usr/local/src/conda/python-3.13.5/Modules/_threadmodule.c", line 343, in thread_run (python)
(Python) File "/home/matveyev/miniforge3/envs/tangodev_13/lib/python3.13/threading.py", line 1014, in _bootstrap
self._bootstrap_inner()
(Python) File "/home/matveyev/miniforge3/envs/tangodev_13/lib/python3.13/threading.py", line 1043, in _bootstrap_inner
self.run()
(Python) File "/home/matveyev/miniforge3/envs/tangodev_13/lib/python3.13/threading.py", line 994, in run
self._target(*self._args, **self._kwargs)
(Python) File "???", line 0, in ???
(C) File "/usr/local/src/conda/python-3.13.5/Objects/call.c", line 381, in PyCFunction_Call (inlined) (python)
(Python) File "/home/matveyev/pytango/tango/test_context.py", line 514, in target
runserver(
(Python) File "???", line 0, in ???
(C) File "/usr/local/src/conda/python-3.13.5/Modules/_functoolsmodule.c", line 353, in partial_call (python)
(Python) File "/home/matveyev/pytango/tango/server.py", line 961, in run_server
return run((cls,), args, **kwargs)
(Python) File "/home/matveyev/pytango/tango/server.py", line 2193, in run
return server_run()
(Python) File "???", line 0, in ???
(C) File "/usr/local/src/conda/python-3.13.5/Modules/_functoolsmodule.c", line 353, in partial_call (python)
(Python) File "/home/matveyev/pytango/tango/server.py", line 2037, in __server_run
worker.run(tango_loop, wait=True)
(Python) File "/home/matveyev/pytango/tango/green.py", line 121, in run
return fn(*args, **kwargs)
(Python) File "/home/matveyev/pytango/tango/server.py", line 2031, in tango_loop
util.server_init()
(C) File "/usr/local/src/conda/python-3.13.5/Objects/methodobject.c", line 540, in cfunction_call (python)
(C) File "/home/matveyev/miniforge3/envs/tangodev_13/lib/python3.13/site-packages/pybind11/include/pybind11/pybind11.h", line 1063, in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (_tango.cpython-313-x86_64-linux-gnu.so)
(C) File "/home/matveyev/miniforge3/envs/tangodev_13/lib/python3.13/site-packages/pybind11/include/pybind11/pybind11.h", line 400, in pybind11::cpp_function::initialize<void (*&)(Tango::Util&, bool), void, Tango::Util&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, char [400], pybind11::arg_v>(void (*&)(Tango::Util&, bool), void (*)(Tango::Util&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [400], pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) (_tango.cpython-313-x86_64-linux-gnu.so)
(C) File "/home/matveyev/miniforge3/envs/tangodev_13/lib/python3.13/site-packages/pybind11/include/pybind11/pybind11.h", line 430, in pybind11::cpp_function::initialize<void (*&)(Tango::Util&, bool), void, Tango::Util&, bool, pybind11::name, pybind11::is_method, pybind11::sibling, char [400], pybind11::arg_v>(void (*&)(Tango::Util&, bool), void (*)(Tango::Util&, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, char const (&) [400], pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::operator()(pybind11::detail::function_call&) const (_tango.cpython-313-x86_64-linux-gnu.so)
(C) File "/home/matveyev/miniforge3/envs/tangodev_13/lib/python3.13/site-packages/pybind11/include/pybind11/cast.h", line 2111, in std::enable_if<std::is_void<void>::value, pybind11::detail::void_type>::type pybind11::detail::argument_loader<Tango::Util&, bool>::call<void, pybind11::detail::void_type, void (*&)(Tango::Util&, bool)>(void (*&)(Tango::Util&, bool)) && (_tango.cpython-313-x86_64-linux-gnu.so)
(C) File "/home/matveyev/miniforge3/envs/tangodev_13/lib/python3.13/site-packages/pybind11/include/pybind11/cast.h", line 2137, in void pybind11::detail::argument_loader<Tango::Util&, bool>::call_impl<void, void (*&)(Tango::Util&, bool), 0ul, 1ul, pybind11::detail::void_type>(void (*&)(Tango::Util&, bool), std::integer_sequence<unsigned long, 0ul, 1ul>, pybind11::detail::void_type&&) && (_tango.cpython-313-x86_64-linux-gnu.so)
(C) File "/home/matveyev/pytango/ext/tango_util.cpp", line 338, in PyUtil::server_init(Tango::Util&, bool) (_tango.cpython-313-x86_64-linux-gnu.so)
(C) File "/home/matveyev/cppTango/src/server/utils.cpp", line 2120, in Tango::Util::server_init(bool) (libtango.so.10.3)
Traceback for thread 568902 [Has the GIL] (most recent call last):
(C) File "/usr/local/src/conda/python-3.13.5/Parser/parser.c", line 36145, in _start (python)
(C) File "../csu/libc-start.c", line 308, in __libc_start_main (libc.so.6)
(C) File "/usr/local/src/conda/python-3.13.5/Modules/main.c", line 829, in Py_BytesMain (python)
(C) File "/usr/local/src/conda/python-3.13.5/Modules/main.c", line 775, in Py_RunMain (python)
(C) File "/usr/local/src/conda/python-3.13.5/Modules/main.c", line 696, in pymain_run_python (inlined) (python)
(C) File "/usr/local/src/conda/python-3.13.5/Modules/main.c", line 429, in pymain_run_file (inlined) (python)
(C) File "/usr/local/src/conda/python-3.13.5/Modules/main.c", line 410, in pymain_run_file_obj (inlined) (python)
(C) File "/usr/local/src/conda/python-3.13.5/Python/pythonrun.c", line 1295, in pyrun_file (python)
(C) File "/usr/local/src/conda/python-3.13.5/Python/pythonrun.c", line 1466, in run_mod (python)
(C) File "/usr/local/src/conda/python-3.13.5/Python/pythonrun.c", line 1381, in run_eval_code_obj (python)
(Python) File "/home/matveyev/pytango/sandbox/server_for_tests.py", line 12, in <module>
with DeviceTestContext(MyDevice, timeout=60000) as dev_ctx2:
(Python) File "???", line 0, in ???
(Python) File "/home/matveyev/pytango/tango/test_context.py", line 938, in __enter__
self.start()
(Python) File "/home/matveyev/pytango/tango/test_context.py", line 658, in start
self.connect()
(Python) File "/home/matveyev/pytango/tango/test_context.py", line 924, in connect
super().connect()
(Python) File "/home/matveyev/pytango/tango/test_context.py", line 663, in connect
self._wait_until_port_is_known()
(Python) File "/home/matveyev/pytango/tango/test_context.py", line 551, in _wait_until_port_is_known
self.port = self._discovered_port_queue.get(timeout=self.timeout)
(Python) File "/home/matveyev/miniforge3/envs/tangodev_13/lib/python3.13/queue.py", line 194, in get
with self.not_empty:
(C) File "/usr/local/src/conda/python-3.13.5/Python/ceval.c", line 1469, in initialize_locals (python)
...Edited by Yury Matveev