Segmentation fault with witness/son node
Getting segmentation fault while running witness/son node on ubuntu 18.04.5 LTS. Has happened twice shortly after the node fully syncs with the network. No other significant processes running on the server besides btc node. Server has 8gb ram and was only utilizing ~1.5 gb prior to this.
stdout:
2021-03-11T23:59:58
3597690ms th_a peerplays_sidechain_plugin.cpp:535 create_son_down_prop ] Sending son down proposal for 1.33.0 from 1.33.6
457ms th_a application.cpp:500 handle_block ] Got block: #38488015 time: 2021-03-12T00:00:00 latency:
457 ms from: eifos-witness irreversible: 38488005 (-10)
457ms th_a db_maint.cpp:1729 process_dividend_ass ] In process_dividend_assets time 2021-03-12T00:00:00
513ms th_a db_maint.cpp:1484 schedule_pending_div ] payout_asset_type: 1.3.0 current_balance: 1 previous_balance: 0
513ms th_a db_maint.cpp:1524 schedule_pending_div ] total_fee_per_asset_in_payout_asset: 0 dividend_data.options: {"next_payout_time":"2021-03-18T16:00:00","payout_interval":2592000,"minimum_fee_percentage":1000,"minimum_distribution_interval":259200,"extensions":[]}
514ms th_a db_maint.cpp:1484 schedule_pending_div ] payout_asset_type: 1.3.1 current_balance: 6 previous_balance: 0
515ms th_a db_maint.cpp:1524 schedule_pending_div ] total_fee_per_asset_in_payout_asset: 0 dividend_data.options: {"next_payout_time":"2021-03-18T16:00:00","payout_interval":2592000,"minimum_fee_percentage":1000,"minimum_distribution_interval":259200,"extensions":[]}
543ms th_a db_maint.cpp:735 update_active_sons ] Active SONs set NOT CHANGED
1043ms th_a peerplays_sidechain_plugin.cpp:392 son_processing ] Scheduled SON: 1.33.4 Now: 2021-03-12T00:00:01
12147ms th_a application.cpp:500 handle_block ] Got block: #38488016 time: 2021-03-12T00:00:12 latency: 147 ms from: houdini-witness irreversible: 38488007 (-9)
12648ms th_a peerplays_sidechain_plugin.cpp:392 son_processing ] Scheduled SON: 1.33.6 Now: 2021-03-12T00:00:13
Segmentation faultoutput of default.log
2021-03-12T00:00:00.457 th_a:invoke handle_block handle_block ] Got block: #38488015 time: 2021-03-12T00:00:00 latency: 457 ms from: eifos-witness irreversible: 38488005 (-10) application.cpp:500
2021-03-12T00:00:00.457 th_a:invoke handle_block process_dividend_ass ] In process_dividend_assets time 2021-03-12T00:00:00 db_maint.cpp:1729
2021-03-12T00:00:00.513 th_a:invoke handle_block schedule_pending_div ] payout_asset_type: 1.3.0 current_balance: 1 previous_balance: 0 db_maint.cpp:1484
2021-03-12T00:00:00.513 th_a:invoke handle_block schedule_pending_div ] total_fee_per_asset_in_payout_asset: 0 dividend_data.options: ["ERROR_WHILE_CONVERTING_VALUE_TO_STRING"] db_maint.cpp:1524
2021-03-12T00:00:00.514 th_a:invoke handle_block schedule_pending_div ] payout_asset_type: 1.3.1 current_balance: 6 previous_balance: 0 db_maint.cpp:1484
2021-03-12T00:00:00.515 th_a:invoke handle_block schedule_pending_div ] total_fee_per_asset_in_payout_asset: 0 dividend_data.options: ["ERROR_WHILE_CONVERTING_VALUE_TO_STRING"] db_maint.cpp:1524
2021-03-12T00:00:00.543 th_a:invoke handle_block update_active_sons ] Active SONs set NOT CHANGED db_maint.cpp:735
2021-03-12T00:00:01.043 th_a:SON Processing son_processing ] Scheduled SON: 1.33.4 Now: 2021-03-12T00:00:01 peerplays_sidechain_plugin.cpp:392
2021-03-12T00:00:12.147 th_a:invoke handle_block handle_block ] Got block: #38488016 time: 2021-03-12T00:00:12 latency: 147 ms from: houdini-witness irreversible: 38488007 (-9) application.cpp:500
2021-03-12T00:00:12.648 th_a:SON Processing son_processing ] Scheduled SON: 1.33.6 Now: 2021-03-12T00:00:13 peerplays_sidechain_plugin.cpp:392output of p2p.log.20210311T120000
node.cpp:1427
2021-03-11T12:59:31.154 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Sending a keepalive message to peer 4.16.58.146:9777 who hasn't sent us any messages in the last 15 seconds node.cpp:1427
2021-03-11T12:59:31.154 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Sending a keepalive message to peer 94.130.8.110:6666 who hasn't sent us any messages in the last 15 seconds node.cpp:1427
2021-03-11T12:59:55.399 p2p:message read_loop on_hello_message ] Received hello message from peer on a different chain: ["ERROR_WHILE_CONVERTING_VALUE_TO_STRING"] node.cpp:1955
2021-03-11T12:59:55.557 p2p:message read_loop on_closing_connectio ] Peer 159.69.13.94:1024 is disconnecting us because: You are on a different chain from me node.cpp:2957
2021-03-11T12:59:55.557 p2p:message read_loop read_loop ] disconnected 0 exception: unspecified
Bad file descriptor
{"message":"Bad file descriptor"}
asio asio.cpp:41 operator()
{"len":16}
p2p stcp_socket.cpp:117 readsome message_oriented_connection.cpp:223
2021-03-11T12:59:55.557 p2p:delayed_peer_deletion_task destroy_connection ] Exception thrown while canceling message_oriented_connection's read_loop, ignoring: ["ERROR_WHILE_CONVERTING_VALUE_TO_STRING"] message_oriented_connection.cpp:312
2021-03-11T12:59:55.557 p2p:delayed_peer_deletion_task destroy_connection ] Exception thrown while canceling message_oriented_connection's read_loop, ignoring: ["ERROR_WHILE_CONVERTING_VALUE_TO_STRING"] message_oriented_connection.cpp:312
2021-03-11T12:59:59.787 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Sending a keepalive message to peer 159.69.145.156:38909 who hasn't sent us any messages in the last 15 seconds node.cpp:1427
2021-03-11T12:59:59.787 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Sending a keepalive message to peer 136.243.8.23:44497 who hasn't sent us any messages in the last 15 seconds node.cpp:1427
2021-03-11T12:59:59.787 p2p:terminate_inactive_connections_loop terminate_inactive_c ] Sending a keepalive message to peer 4.16.58.149:9777 who hasn't sent us any messages in the last 15 seconds node.cpp:1427output of rpc.log:
2021-03-11T20:44:59.875 th_a:? operator() ] msg->get_payload(): {"method":"notice","params":[0,["01cc2b06297704f13ea49ea89586a5b24f3e87f1"]]} websocket.cpp:417
2021-03-11T20:44:59.877 th_a:? operator() ] msg->get_payload(): {"method":"notice","params":[0,["01cc2b07426842e889ac25b3ff152837c3c8ecae"]]} websocket.cpp:417
2021-03-11T20:44:59.877 th_a:? operator() ] msg->get_payload(): {"id":32,"jsonrpc":"2.0","result":null} websocket.cpp:417Let me know if any more information is needed.
Edit: Some additional information output of df -h
df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 798M 724K 797M 1% /run
/dev/sda1 912G 368G 499G 43% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda2 465M 60M 377M 14% /boot
tmpfs 798M 0 798M 0% /run/user/0
tmpfs 798M 0 798M 0% /run/user/1000output of free -h
free -h
total used free shared buff/cache available
Mem: 7.8G 1.2G 218M 268K 6.3G 6.3G
Swap: 3.8G 115M 3.7GThis error is able to be reproduced multiple times shortly after the node syncs with the network. Ram utilization is around 2GB of total of 8GB when the fault occurs.
Edit 2, additional info: After removing sidechain_plugin from the config, the witness works perfectly, so it is definitely something with the sidechain plugin that is triggering the fault.
Edited by jburr ppy