Crash on read attribute combined with pushing events (SerialModel.NO_SYNC)
We have a tango device server crashing in read attribute. Original crash was on cpptango 9.3.5. We updated to 9.5.0 and got the same issue.
The server is pushing change events but the crash occurs randomly when reading some dynamic attributes. I managed to reproduce locally by writing a device sever using the same logic: https://gitlab.com/beenje/pytango-playground/-/blob/main/read_dyn_attr_crash.py?ref_type=heads
It should be easy to reproduce with that example running python read_dyn_attr_crash.py test
.
It's using DeviceTestContext
, so no need for a database. By default it reads attributes every 2 seconds and crashes quite fast like that on my machine. One can pass --sleep X
to change this period.
It's also possible to run the server and client separately (read_dyn_attr_crash.py server
/ read_dyn_attr_crash.py client
).
Here is the stack trace I get:
(gdb) bt
#0 _CORBA_Sequence<double>::get_buffer (orphan=<optimized out>, this=<optimized out>) at /usr/local/src/conda-prefix/include/omniORB4/seqTemplatedecls.h:142
#1 Tango::(anonymous namespace)::data_in_object<Tango::DevVarDoubleArray> (att=..., aid=..., index=<optimized out>, del_seq=<optimized out>)
at /usr/local/src/conda/cpptango-9.5.0/src/server/device.cpp:356
#2 0x00007f79c25cb7ee in Tango::Device_3Impl::read_attributes_no_except (this=this@entry=0x7f799c00ff90, names=..., aid=..., second_try=second_try@entry=true, idx=...)
at /usr/local/src/conda/cpptango-9.5.0/src/server/device_3.cpp:1016
#3 0x00007f79c25e0ff0 in Tango::Device_5Impl::read_attributes_5 (this=0x7f799c00ff90, names=..., source=<optimized out>, cl_id=...)
at /usr/local/src/conda/cpptango-9.5.0/src/server/device_5.cpp:357
#4 0x00007f79c2521c8a in _0RL_lcfn_6fe2f94a21a10053_84000000 (cd=0x7f79b969ea50, svnt=<optimized out>) at /usr/local/src/conda/cpptango-9.5.0/build/src/include/tango/idl/tangoSK.cpp:6494
#5 0x00007f79c2589192 in omniCallDescriptor::interceptedCall (servant=0x7f799c0106b8, this=0x7f79b969ea50) at /usr/local/src/conda-prefix/include/omniORB4/callDescriptor.h:159
#6 Tango::client_call_interceptor (d=0x7f79b969ea50, s=0x7f799c0106b8) at /usr/local/src/conda/cpptango-9.5.0/src/server/blackbox.cpp:111
#7 0x00007f79c211c8b9 in omniCallHandle::upcall(omniServant*, omniCallDescriptor&) ()
from /mxn/home/benber/.conda/envs/pytango95/lib/python3.11/site-packages/tango/../../../libomniORB4.so.3
#8 0x00007f79c252623e in Tango::_impl_Device_5::_dispatch (this=<optimized out>, _handle=...) at /usr/local/src/conda/cpptango-9.5.0/build/src/include/tango/idl/tangoSK.cpp:7416
#9 0x00007f79c2115786 in omni::omniOrbPOA::dispatch(omniCallHandle&, omniLocalIdentity*) ()
from /mxn/home/benber/.conda/envs/pytango95/lib/python3.11/site-packages/tango/../../../libomniORB4.so.3
#10 0x00007f79c20f5936 in omniLocalIdentity::dispatch(omniCallHandle&) () from /mxn/home/benber/.conda/envs/pytango95/lib/python3.11/site-packages/tango/../../../libomniORB4.so.3
#11 0x00007f79c2139ff4 in omni::GIOP_S::handleRequest() () from /mxn/home/benber/.conda/envs/pytango95/lib/python3.11/site-packages/tango/../../../libomniORB4.so.3
#12 0x00007f79c213a5d9 in omni::GIOP_S::dispatcher() () from /mxn/home/benber/.conda/envs/pytango95/lib/python3.11/site-packages/tango/../../../libomniORB4.so.3
#13 0x00007f79c2137c96 in omni::giopWorker::execute() () from /mxn/home/benber/.conda/envs/pytango95/lib/python3.11/site-packages/tango/../../../libomniORB4.so.3
#14 0x00007f79c20e918a in omniAsyncWorker::real_run() () from /mxn/home/benber/.conda/envs/pytango95/lib/python3.11/site-packages/tango/../../../libomniORB4.so.3
#15 0x00007f79c20e962e in omniAsyncPoolServer::workerRun(omniAsyncWorker*) () from /mxn/home/benber/.conda/envs/pytango95/lib/python3.11/site-packages/tango/../../../libomniORB4.so.3
#16 0x00007f79c20e8c09 in omniAsyncWorker::mid_run() () from /mxn/home/benber/.conda/envs/pytango95/lib/python3.11/site-packages/tango/../../../libomniORB4.so.3
#17 0x00007f79c20e9094 in omniAsyncWorker::run(void*) () from /mxn/home/benber/.conda/envs/pytango95/lib/python3.11/site-packages/tango/../../../libomniORB4.so.3
#18 0x00007f79c201e035 in omni_thread_wrapper () from /mxn/home/benber/.conda/envs/pytango95/lib/python3.11/site-packages/tango/../../../libomnithread.so.4
#19 0x00007f79ee1ac1ca in start_thread () from /lib64/libpthread.so.0
#20 0x00007f79ed68ee73 in clone () from /lib64/libc.so.6
(gdb) sel 2
(gdb) info locals
w_type = Tango::READ
atsm = <optimized out>
att = @0x562585214450: {_vptr.Attribute = 0x7f79c28df2e0 <vtable for Tango::Attribute+16>, value_flag = true, when = {tv_sec = 1701342347, tv_usec = 484437, tv_nsec = 55}, date = true,
quality = Tango::ATTR_VALID, name = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<std::__new_allocator<char>> = {<No data fields>}, <No data fields>},
_M_p = 0x562585214480 "H22"}, _M_string_length = 3, {_M_local_buf = "H22\000%V\000\000\360\022`\270y\177\000", _M_allocated_capacity = 94716917068360}}, writable = Tango::READ,
data_type = 5, data_format = Tango::SCALAR, max_x = 1, max_y = 0, label = {static npos = 18446744073709551615,
_M_dataplus = {<std::allocator<char>> = {<std::__new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x5625852144c8 "H22"}, _M_string_length = 3, {
_M_local_buf = "H22\000%V\000\000\020f`\270y\177\000", _M_allocated_capacity = 94716917068360}}, description = {static npos = 18446744073709551615,
...
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f79b969f700 (LWP 3693995) _CORBA_Sequence<double>::get_buffer (orphan=<optimized out>, this=<optimized out>)
at /usr/local/src/conda-prefix/include/omniORB4/seqTemplatedecls.h:142
2 Thread 0x7f79bb6a3700 (LWP 3693991) 0x00007f79ed783e87 in epoll_wait () from /lib64/libc.so.6
3 Thread 0x7f79d3f1e700 (LWP 3693988) 0x00007f79ed6a48ac in sigtimedwait () from /lib64/libc.so.6
4 Thread 0x7f79b9ea0700 (LWP 3693994) 0x00007f79ee1b245c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
5 Thread 0x7f79b8e9e700 (LWP 3693996) 0x00007f79ee1b4da6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
6 Thread 0x7f79ba6a1700 (LWP 3693993) 0x00007f79ee1b27aa in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
7 Thread 0x7f79dc71f700 (LWP 3693989) 0x00007f79ed778f41 in poll () from /lib64/libc.so.6
8 Thread 0x7f79b37fe700 (LWP 3693998) 0x00007f79ee1b4da6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
9 Thread 0x7f79baea2700 (LWP 3693992) 0x00007f79ed783e87 in epoll_wait () from /lib64/libc.so.6
10 Thread 0x7f79dcf20700 (LWP 3693990) 0x00007f79ed778f41 in poll () from /lib64/libc.so.6
11 Thread 0x7f79b2ffd700 (LWP 3693999) 0x00007f79ee1b4da6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
12 Thread 0x7f79ee5de740 (LWP 3693987) 0x00007f79ed783e87 in epoll_wait () from /lib64/libc.so.6
13 Thread 0x7f79b3fff700 (LWP 3693997) 0x00007f79ee1b4da6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
Note that if I comment out line 53 and don't send events, there is no crash...