I have a coredump of a python 3.10 process using pytango 10.1.0 which has crashed with the following pybind11 exception being thrown from a thread that creates and "quickly" deletes a `tnago.Database` object about once a second: ``` "pybind11_object_dealloc(): Tried to deallocate unregistered instance!" ``` This is thrown from [`pybind11::detail::clear_instance`](https://github.com/pybind/pybind11/blob/f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8/include/pybind11/detail/class.h#L453) from the Python `frame_dealloc` method when we returning from the function that holds the reference to the `tango.Database` object. To me it looks like we are running into https://github.com/pybind/pybind11/issues/5473 as we are using `py::call_guard<py::gil_scoped_release>()` for the `init()` of `Database`. I've tried a modified version of the reproduction from the pybind11 issue with no luck I'm afraid so maybe this isn't the issue, however, I struggle to see how this can be anything but a pybind11 bug as this registration is something that should be managed by pybind11. When I was trying to reproduce this I tried adding a latency to my databaseds docker container using `tc qdisc` to replicate the `std::this_thread::sleep_for` they have in their reproduction, but this didn't help either. Any ideas how we can reproduce this to verify that the fix they propose in the pybind11 issue works for us? <details> <summary>gdb session</summary> ``` Core was generated by `python /data/mccs_master_leaf_node.py 01 -ORBendPoint giop:tcp::45450 -ORBendPo'. Program terminated with signal SIGABRT, Aborted. warning: Section `.reg-xstate/81' in core file too small. #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=126320323311168) at ./nptl/pthread_kill.c:44 Download failed: Invalid argument. Continuing without source file ./nptl/./nptl/pthread_kill.c. 44 ./nptl/pthread_kill.c: No such file or directory. [Current thread is 1 (Thread 0x72e33dffb640 (LWP 81))] (gdb) bt #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=126320323311168) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=126320323311168) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=126320323311168, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x000072e3f2c20476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x000072e3f2c067f3 in __GI_abort () at ./stdlib/abort.c:79 #5 0x000072e3f0d68b9e in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x000072e3f0d7420c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x000072e3f0d74277 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6 #8 0x000072e3f0d744d8 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6 #9 0x000072e3f22aaf78 in pybind11::pybind11_fail ( reason=reason@entry=0x72e3f2639a98 "pybind11_object_dealloc(): Tried to deallocate unregistered instance!") at /tmp/build-env-uyc7_13l/lib/python3.10/site-packages/pybind11/include/pybind11/detail/common.h:1103 #10 0x000072e3f22fc74a in pybind11::detail::clear_instance (self=0x72e37dde3420) at /tmp/build-env-uyc7_13l/lib/python3.10/site-packages/pybind11/include/pybind11/detail/class.h:453 #11 0x000072e3f22fc7b1 in pybind11::detail::pybind11_object_dealloc (self=0x72e37dde3420) at /tmp/build-env-uyc7_13l/lib/python3.10/site-packages/pybind11/include/pybind11/detail/class.h:493 #12 0x0000622de20e1390 in _Py_Dealloc (op=<optimized out>) at ../Objects/object.c:2301 #13 _Py_DECREF (op=<optimized out>) at ../Include/object.h:500 #14 frame_dealloc (f=0x72e32c000cb0) at ../Objects/frameobject.c:591 #15 0x0000622de20eb6e3 in _Py_Dealloc (op=0x72e32c000cb0) at ../Objects/object.c:2301 #16 _Py_DECREF (op=0x72e32c000cb0) at ../Include/object.h:500 #17 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x72e386995b50, tstate=0x622dffe9bdc0) at ../Python/ceval.c:5080 #18 _PyFunction_Vectorcall (func=0x72e386995b40, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342 #19 0x0000622de20d5a77 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x72e3869e9150, callable=0x72e386995b40, tstate=0x622dffe9bdc0) at ../Include/cpython/abstract.h:114 #20 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x72e3869e9150, callable=0x72e386995b40) at ../Include/cpython/abstract.h:123 #21 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x72e33dffa700, tstate=<optimized out>) at ../Python/ceval.c:5893 #22 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4198 #23 0x0000622de20f8b81 in _PyEval_EvalFrame (throwflag=0, f=0x72e3869e8fc0, tstate=0x622dffe9bdc0) at ../Include/internal/pycore_ceval.h:46 #24 _PyEval_Vector (kwnames=0x0, argcount=1, args=0x72e33dffa7a8, locals=0x0, con=0x72e386995e20, tstate=0x622dffe9bdc0) at ../Python/ceval.c:5067 #25 _PyFunction_Vectorcall (kwnames=0x0, nargsf=1, stack=0x72e33dffa7a8, func=0x72e386995e10) at ../Objects/call.c:342 #26 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x72e33dffa7a8, callable=0x72e386995e10, tstate=0x622dffe9bdc0) at ../Include/cpython/abstract.h:114 #27 method_vectorcall (method=<optimized out>, args=0x72e3f2a84088, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/classobject.c:61 #28 0x0000622de20d7ceb in do_call_core (kwdict=0x72e383764240, callargs=0x72e3f2a84070, func=0x72e38378d640, trace_info=0x72e33dffa940, tstate=<optimized out>) at ../Python/ceval.c:5945 #29 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4277 #30 0x0000622de20eb6ac in _PyEval_EvalFrame (throwflag=0, f=0x72e3837018c0, tstate=0x622dffe9bdc0) at ../Include/internal/pycore_ceval.h:46 #31 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x72e3e4d24710, tstate=0x622dffe9bdc0) at ../Python/ceval.c:5067 #32 _PyFunction_Vectorcall (func=0x72e3e4d24700, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342 #33 0x0000622de20d5a77 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x72e3869e8f40, callable=0x72e3e4d24700, tstate=0x622dffe9bdc0) at ../Include/cpython/abstract.h:114 #34 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x72e3869e8f40, callable=0x72e3e4d24700) at ../Include/cpython/abstract.h:123 #35 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x72e33dffab20, tstate=<optimized out>) at ../Python/ceval.c:5893 #36 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4198 #37 0x0000622de20eb6ac in _PyEval_EvalFrame (throwflag=0, f=0x72e3869e8dd0, tstate=0x622dffe9bdc0) at ../Include/internal/pycore_ceval.h:46 #38 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=<optimized out>, locals=0x0, con=0x72e3e4d249e0, tstate=0x622dffe9bdc0) at ../Python/ceval.c:5067 #39 _PyFunction_Vectorcall (func=0x72e3e4d249d0, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:342 #40 0x0000622de20d5a77 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x72e3869d70f0, callable=0x72e3e4d249d0, tstate=0x622dffe9bdc0) at ../Include/cpython/abstract.h:114 #41 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x72e3869d70f0, callable=0x72e3e4d249d0) at ../Include/cpython/abstract.h:123 #42 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x72e33dffad00, tstate=<optimized out>) at ../Python/ceval.c:5893 #43 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:4198 #44 0x0000622de20f8b81 in _PyEval_EvalFrame (throwflag=0, f=0x72e3869d6f80, tstate=0x622dffe9bdc0) at ../Include/internal/pycore_ceval.h:46 #45 _PyEval_Vector (kwnames=0x0, argcount=1, args=0x72e33dffada8, locals=0x0, con=0x72e3e4d247a0, --Type <RET> for more, q to quit, c to continue without paging--c tstate=0x622dffe9bdc0) at ../Python/ceval.c:5067 #46 _PyFunction_Vectorcall (kwnames=0x0, nargsf=1, stack=0x72e33dffada8, func=0x72e3e4d24790) at ../Objects/call.c:342 #47 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x72e33dffada8, callable=0x72e3e4d24790, tstate=0x622dffe9bdc0) at ../Include/cpython/abstract.h:114 #48 method_vectorcall (method=<optimized out>, args=0x72e3f2a84088, nargsf=<optimized out>, kwnames=<optimized out>) at ../Objects/classobject.c:61 #49 0x0000622de220d9fa in thread_run (boot_raw=0x72e3b01671e0) at ../Modules/_threadmodule.c:1100 #50 0x0000622de2203598 in pythread_wrapper (arg=<optimized out>) at ../Python/thread_pthread.h:253 #51 0x000072e3f2c72ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #52 0x000072e3f2d04850 in __closefrom_fallback (from=482171712, dirfd_fallback=<optimized out>) at ../sysdeps/unix/sysv/linux/closefrom_fallback.c:43 #53 0x0000000000000000 in ?? () (gdb) frame 14 #14 frame_dealloc (f=0x72e32c000cb0) at ../Objects/frameobject.c:591 591 in ../Objects/frameobject.c (gdb) p *f $4 = {ob_base = {ob_base = {ob_refcnt = 0, ob_type = 0x622de24d5ec0 <PyFrame_Type>}, ob_size = 21}, f_back = 0x72e3869e8fc0, f_code = 0x72e3869b9420, f_builtins = 0x72e3f2b1b6c0, f_globals = 0x72e3869a84c0, f_locals = 0x0, f_valuestack = 0x72e32c000e68, f_trace = 0x0, f_stackdepth = 0, f_trace_lines = 1 '\001', f_trace_opcodes = 0 '\000', f_gen = 0x0, f_lasti = 299, f_lineno = 0, f_iblock = 0, f_state = 1 '\001', f_blockstack = {{b_type = 122, b_handler = 80, b_level = 0}, {b_type = 0, b_handler = 0, b_level = 0} <repeats 19 times>}, f_localsplus = {0x0}} ``` </details>