after stopping witness node peerplays02 other witness nodes crashed.
Steps:
- After stopping witness node peerplays02 other witness nodes crashed.
kiran@warpspeed:~/PBSA/06012023/peerplays-utils/peerplays-qa-environment$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f7ba51b8a3fd peerplays05 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8094->8090/tcp, :::8094->8090/tcp, 0.0.0.0:9781->9777/tcp, :::9781->9777/tcp peerplays-qa-environment_peerplays05_1
a413e77f303b peerplays10 "./witness_node -d .…" 45 minutes ago Up 8 seconds 22/tcp, 0.0.0.0:8099->8090/tcp, :::8099->8090/tcp, 0.0.0.0:9786->9777/tcp, :::9786->9777/tcp peerplays-qa-environment_peerplays10_1
16b87407ac40 peerplays14 "./witness_node -d .…" 45 minutes ago Up 7 seconds 22/tcp, 0.0.0.0:8103->8090/tcp, :::8103->8090/tcp, 0.0.0.0:9790->9777/tcp, :::9790->9777/tcp peerplays-qa-environment_peerplays14_1
fc00fe7ec784 peerplays06 "./witness_node -d .…" 45 minutes ago Up 8 seconds 22/tcp, 0.0.0.0:8095->8090/tcp, :::8095->8090/tcp, 0.0.0.0:9782->9777/tcp, :::9782->9777/tcp peerplays-qa-environment_peerplays06_1
f366d30cf234 peerplays12 "./witness_node -d .…" 45 minutes ago Up 19 seconds 22/tcp, 0.0.0.0:8101->8090/tcp, :::8101->8090/tcp, 0.0.0.0:9788->9777/tcp, :::9788->9777/tcp peerplays-qa-environment_peerplays12_1
5eb751b3e493 peerplays07 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8096->8090/tcp, :::8096->8090/tcp, 0.0.0.0:9783->9777/tcp, :::9783->9777/tcp peerplays-qa-environment_peerplays07_1
c736743f669b peerplays04 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8093->8090/tcp, :::8093->8090/tcp, 0.0.0.0:9780->9777/tcp, :::9780->9777/tcp peerplays-qa-environment_peerplays04_1
a781598646cd peerplays03 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8092->8090/tcp, :::8092->8090/tcp, 0.0.0.0:9779->9777/tcp, :::9779->9777/tcp peerplays-qa-environment_peerplays03_1
26424f96836e peerplays08 "./witness_node -d .…" 45 minutes ago Restarting (139) Less than a second ago peerplays-qa-environment_peerplays08_1
45b8579384e4 peerplays01 "./witness_node -d .…" 45 minutes ago Up 45 minutes 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp, 22/tcp, 0.0.0.0:9777->9777/tcp, :::9777->9777/tcp peerplays-qa-environment_peerplays01_1
6c11ee69e16b peerplays11 "./witness_node -d .…" 45 minutes ago Up 20 seconds 22/tcp, 0.0.0.0:8100->8090/tcp, :::8100->8090/tcp, 0.0.0.0:9787->9777/tcp, :::9787->9777/tcp peerplays-qa-environment_peerplays11_1
70d9996e2e0d peerplays15 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8104->8090/tcp, :::8104->8090/tcp, 0.0.0.0:9791->9777/tcp, :::9791->9777/tcp peerplays-qa-environment_peerplays15_1
2c084dd3c8cf peerplays09 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8098->8090/tcp, :::8098->8090/tcp, 0.0.0.0:9785->9777/tcp, :::9785->9777/tcp peerplays-qa-environment_peerplays09_1
0d3b2dbcdb29 peerplays16 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8105->8090/tcp, :::8105->8090/tcp, 0.0.0.0:9792->9777/tcp, :::9792->9777/tcp peerplays-qa-environment_peerplays16_1
18d4edab2023 peerplays02 "./witness_node -d .…" 45 minutes ago Exited (137) 32 minutes ago peerplays-qa-environment_peerplays02_1
e2c6788d5c73 peerplays13 "./witness_node -d .…" 45 minutes ago Up 45 minutes 22/tcp, 0.0.0.0:8102->8090/tcp, :::8102->8090/tcp, 0.0.0.0:9789->9777/tcp, :::9789->9777/tcp peerplays-qa-environment_peerplays13_1
e694f0b5083c peerplays-base "bash" 45 minutes ago Exited (0) 45 minutes ago peerplays-qa-environment_peerplays-base_1
40f563d58db9 hive-for-peerplays "./hived -d ./hived_…" 59 minutes ago Up 59 minutes 22/tcp, 0.0.0.0:28090-28091->28090-28091/tcp, :::28090-28091->28090-28091/tcp peerplays-qa-environment_hive-for-peerplays_1
7eeb00febe10 ethereum-for-peerplays "./geth --config ./n…" About an hour ago Up About an hour 0.0.0.0:6060->6060/tcp, :::6060->6060/tcp, 22/tcp, 0.0.0.0:8545->8545/tcp, :::8545->8545/tcp, 8546/tcp, 0.0.0.0:30303->30303/tcp, :::30303->30303/tcp peerplays-qa-environment_ethereum-for-peerplays_1
467391e4c22f libbitcoinserver-for-peerplays "/bin/sh runserver.sh" About an hour ago Up About an hour 0.0.0.0:9091->9091/tcp, :::9091->9091/tcp, 22/tcp, 0.0.0.0:9093->9093/tcp, :::9093->9093/tcp peerplays-qa-environment_libbitcoinserver-for-peerplays_1
604f2e13e799 bitcoin-for-peerplays "./bitcoind -datadir…" About an hour ago Up About an hour 0.0.0.0:8332->8332/tcp, :::8332->8332/tcp, 0.0.0.0:11111->11111/tcp, :::11111->11111/tcp, 22/tcp, 0.0.0.0:18444->18444/tcp, :::18444->18444/tcp peerplays-qa-environment_bitcoin-for-peerplays_1
0a99747528d5 ubuntu-for-peerplays "bash" About an hour ago Exited (0) 45 minutes ago peerplays-qa-environment_ubuntu-for-peerplays_1
ad0ee593076b e28a50f651f9 "bash" 4 days ago Exited (0) 4 days ago clever_hodgkin
2b1ca83f2f7c ubuntu:18.04 "bash" 4 days ago Exited (0) 4 days ago inspiring_tharp
kiran@warpspeed:~/PBSA/06012023/peerplays-utils/peerplays-qa-environment$ kiran@warpspeed:/home$ sudo coredumpctl debug 4999
PID: 4999 (witness_node)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Fri 2023-01-13 00:17:36 +0530 (9min ago)
Command Line: ./witness_node -d ./witness_node_data_dir --replay-blockchain
Executable: /home/peerplays/src/peerplays/build/programs/witness_node/witness_node
Control Group: /docker/16b87407ac4019e43c2c16987ae405a3af7d52c9256235aded38ac02c6acf5ff
Slice: -.slice
Boot ID: 4ced3cb6f68d4f38b5ad5574dd873a6e
Machine ID: a388625228614d18b5373ad1be39a93f
Hostname: warpspeed
Storage: /var/lib/systemd/coredump/core.witness_node.0.4ced3cb6f68d4f38b5ad5574dd873a6e.4999.1673549256000000000000.lz4
Message: Process 4999 (witness_node) of user 0 dumped core.
Stack trace of thread 15:
#0 0x00007f4f8c6ba941 __GI_abort (libc.so.6 + 0x22941)
#1 0x00007f4f8c6ba729 __assert_fail_base (libc.so.6 + 0x22729)
#2 0x00007f4f8c6cbfd6 __GI___assert_fail (libc.so.6 + 0x33fd6)
#3 0x000055cea30ef588 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x57cb588)
#4 0x000055cea310cb13 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x57e8b13)
#5 0x000055cea3259c26 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5935c26)
#6 0x000055cea326071b n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x593c71b)
#7 0x000055cea3260031 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x593c031)
#8 0x000055cea3265a0e n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5941a0e)
#9 0x000055cea45b7138 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c93138)
#10 0x000055cea45b70b4 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c930b4)
#11 0x000055cea45a8c62 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c84c62)
#12 0x000055cea45a9011 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c85011)
#13 0x000055cea45a85e1 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c845e1)
#14 0x000055cea481d96f n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6ef996f)
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/peerplays/src/peerplays/build/programs/witness_node/witness_node...
[New LWP 15]
[New LWP 9]
[New LWP 12]
[New LWP 13]
[New LWP 7]
[New LWP 19]
[New LWP 16]
[New LWP 2442]
[New LWP 18]
[New LWP 10]
[New LWP 17]
[New LWP 2441]
[New LWP 2443]
[New LWP 8]
[New LWP 1]
[New LWP 11]
[New LWP 14]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./witness_node -d ./witness_node_data_dir --replay-blockchain'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __GI_abort () at abort.c:107
107 abort.c: No such file or directory.
[Current thread is 1 (Thread 0x7f4f727fc700 (LWP 15))]
(gdb) bt
#0 __GI_abort () at abort.c:107
#1 0x00007f4f8c6ba729 in __assert_fail_base (fmt=0x7f4f8c850588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55cea4cb8458 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x55cea4cb69e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1654,
function=<optimized out>) at assert.c:92
#2 0x00007f4f8c6cbfd6 in __GI___assert_fail (assertion=0x55cea4cb8458 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x55cea4cb69e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1654,
function=0x55cea4cb8360 "void graphene::net::detail::node_impl::schedule_peer_for_deletion(const peer_connection_ptr&)") at assert.c:101
#3 0x000055cea30ef588 in graphene::net::detail::node_impl::schedule_peer_for_deletion (this=0x55cea6cae5a0, peer_to_delete=std::shared_ptr<class graphene::net::peer_connection> (use count 4, weak count 2) = {...}) at /home/peerplays/src/peerplays/libraries/net/node.cpp:1654
#4 0x000055cea310cb13 in graphene::net::detail::node_impl::on_connection_closed (this=0x55cea6cae5a0, originating_peer=0x7f4f5c07ded0) at /home/peerplays/src/peerplays/libraries/net/node.cpp:3106
#5 0x000055cea3259c26 in graphene::net::peer_connection::on_connection_closed (this=0x7f4f5c07ded0, originating_connection=0x7f4f5c07df00) at /home/peerplays/src/peerplays/libraries/net/peer_connection.cpp:283
#6 0x000055cea326071b in graphene::net::detail::message_oriented_connection_impl::read_loop (this=0x7f4f5c07e2c0) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:241
#7 0x000055cea3260031 in graphene::net::detail::message_oriented_connection_impl::<lambda()>::operator()(void) const (__closure=0x7f4f5c159c08) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:134
#8 0x000055cea3265a0e in fc::detail::void_functor_run<graphene::net::detail::message_oriented_connection_impl::connect_to(const fc::ip::endpoint&)::<lambda()> >::run(void *, void *) (functor=0x7f4f5c159c08, prom=0x7f4f5c159c00) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/task.hpp:83
#9 0x000055cea45b7138 in fc::task_base::run_impl (this=0x7f4f5c159c10) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:43
#10 0x000055cea45b70b4 in fc::task_base::run (this=0x7f4f5c159c10) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:32
#11 0x000055cea45a8c62 in fc::thread_d::run_next_task (this=0x7f4f5c000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:520
#12 0x000055cea45a9011 in fc::thread_d::process_tasks (this=0x7f4f5c000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:568
#13 0x000055cea45a85e1 in fc::thread_d::start_process_tasks (my=...) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:499
#14 0x000055cea481d96f in make_fcontext ()
#15 0x0000000000000000 in ?? ()
(gdb) kiran@warpspeed:/home$ sudo coredumpctl debug 1935140
PID: 1935140 (witness_node)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Fri 2023-01-13 20:10:39 +0530 (26min ago)
Command Line: ./witness_node -d ./witness_node_data_dir --replay-blockchain
Executable: /home/peerplays/src/peerplays/build/programs/witness_node/witness_node
Control Group: /docker/2c084dd3c8cfb2d4b01898de6e02665b9594bccd75aabd1d080ff4db2ee81847
Slice: -.slice
Boot ID: 4ced3cb6f68d4f38b5ad5574dd873a6e
Machine ID: a388625228614d18b5373ad1be39a93f
Hostname: warpspeed
Storage: /var/lib/systemd/coredump/core.witness_node.0.4ced3cb6f68d4f38b5ad5574dd873a6e.1935140.1673620839000000000000.lz4
Message: Process 1935140 (witness_node) of user 0 dumped core.
Stack trace of thread 15:
#0 0x00007fe937fb3941 __GI_abort (libc.so.6 + 0x22941)
#1 0x00007fe937fb3729 __assert_fail_base (libc.so.6 + 0x22729)
#2 0x00007fe937fc4fd6 __GI___assert_fail (libc.so.6 + 0x33fd6)
#3 0x00005634786d4588 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x57cb588)
#4 0x00005634786f1b13 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x57e8b13)
#5 0x000056347883ec26 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5935c26)
#6 0x000056347884571b n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x593c71b)
#7 0x0000563478845031 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x593c031)
#8 0x000056347884aa0e n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x5941a0e)
#9 0x0000563479b9c138 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c93138)
#10 0x0000563479b9c0b4 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c930b4)
#11 0x0000563479b8dc62 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c84c62)
#12 0x0000563479b8e011 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c85011)
#13 0x0000563479b8d5e1 n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6c845e1)
#14 0x0000563479e0296f n/a (/home/peerplays/src/peerplays/build/programs/witness_node/witness_node + 0x6ef996f)
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/peerplays/src/peerplays/build/programs/witness_node/witness_node...
[New LWP 15]
[New LWP 10]
[New LWP 8]
[New LWP 9]
[New LWP 11]
[New LWP 14]
[New LWP 16]
[New LWP 1]
[New LWP 7]
[New LWP 13]
[New LWP 19]
[New LWP 12]
[New LWP 17]
[New LWP 18]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./witness_node -d ./witness_node_data_dir --replay-blockchain'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __GI_abort () at abort.c:107
107 abort.c: No such file or directory.
[Current thread is 1 (Thread 0x7fe91dffb700 (LWP 15))]
(gdb) bt
#0 __GI_abort () at abort.c:107
#1 0x00007fe937fb3729 in __assert_fail_base (fmt=0x7fe938149588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x56347a29d458 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x56347a29b9e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1654,
function=<optimized out>) at assert.c:92
#2 0x00007fe937fc4fd6 in __GI___assert_fail (assertion=0x56347a29d458 "_closing_connections.find(peer_to_delete) == _closing_connections.end()", file=0x56347a29b9e8 "/home/peerplays/src/peerplays/libraries/net/node.cpp", line=1654,
function=0x56347a29d360 "void graphene::net::detail::node_impl::schedule_peer_for_deletion(const peer_connection_ptr&)") at assert.c:101
#3 0x00005634786d4588 in graphene::net::detail::node_impl::schedule_peer_for_deletion (this=0x56347ed06780, peer_to_delete=std::shared_ptr<class graphene::net::peer_connection> (use count 4, weak count 2) = {...}) at /home/peerplays/src/peerplays/libraries/net/node.cpp:1654
#4 0x00005634786f1b13 in graphene::net::detail::node_impl::on_connection_closed (this=0x56347ed06780, originating_peer=0x7fe908133700) at /home/peerplays/src/peerplays/libraries/net/node.cpp:3106
#5 0x000056347883ec26 in graphene::net::peer_connection::on_connection_closed (this=0x7fe908133700, originating_connection=0x7fe908133730) at /home/peerplays/src/peerplays/libraries/net/peer_connection.cpp:283
#6 0x000056347884571b in graphene::net::detail::message_oriented_connection_impl::read_loop (this=0x7fe908133af0) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:241
#7 0x0000563478845031 in graphene::net::detail::message_oriented_connection_impl::<lambda()>::operator()(void) const (__closure=0x7fe90814cad8) at /home/peerplays/src/peerplays/libraries/net/message_oriented_connection.cpp:134
#8 0x000056347884aa0e in fc::detail::void_functor_run<graphene::net::detail::message_oriented_connection_impl::connect_to(const fc::ip::endpoint&)::<lambda()> >::run(void *, void *) (functor=0x7fe90814cad8, prom=0x7fe90814cad0) at /home/peerplays/src/peerplays/libraries/fc/include/fc/thread/task.hpp:83
#9 0x0000563479b9c138 in fc::task_base::run_impl (this=0x7fe90814cae0) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:43
#10 0x0000563479b9c0b4 in fc::task_base::run (this=0x7fe90814cae0) at /home/peerplays/src/peerplays/libraries/fc/src/thread/task.cpp:32
#11 0x0000563479b8dc62 in fc::thread_d::run_next_task (this=0x7fe908000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:520
#12 0x0000563479b8e011 in fc::thread_d::process_tasks (this=0x7fe908000b60) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:568
#13 0x0000563479b8d5e1 in fc::thread_d::start_process_tasks (my=...) at /home/peerplays/src/peerplays/libraries/fc/src/thread/thread_d.hpp:499
#14 0x0000563479e0296f in make_fcontext ()
#15 0x0000000000000000 in ?? ()
(gdb)
Logs at: https://gitlab.com/PBSA/documentation/working-documents/-/blob/develop/501/crash.tar.gz
Edited by Kiran Kumar