investigate BEATRICE halt
Beatrice stopped making blocks on July 4. LIB is 16545027 (2020-07-04T00:59:30) from blckchnd-testnet
After maintenance of 2020-07-04T01:00:00, LIB stop increasing. Last "Got block" at 2020-07-04T01:00:48, and no more blocks. All witness_node continue running.
Part of logs. No blocks after:
3597023ms th_a application.cpp:499 handle_block ] Got block: #16545034 time: 2020-07-04T00:59:57 latency: 23 ms from: roelandp irreversible: 16545026 (-8)
3104ms th_a application.cpp:499 handle_block ] Got block: #16545035 time: 2020-07-04T01:00:03 latency: 104 ms from: houdini-testnet irreversible: 16545027 (-8)
3105ms th_a db_maint.cpp:1321 process_dividend_ass ] In process_dividend_assets time 2020-07-04T01:00:03
3349ms th_a db_maint.cpp:1076 schedule_pending_div ] payout_asset_type: 1.3.1 current_balance: 1 previous_balance: 0
3349ms th_a db_maint.cpp:1116 schedule_pending_div ] total_fee_per_asset_in_payout_asset: 0 dividend_data.options: {"next_payout_time":"2020-07-12T20:00:00","payout_interval":2592000,"minimum_fee_percentage":1000,"minimum_distribution_interval":259200,"extensions":[]}
3485ms th_a application.cpp:499 handle_block ] Got block: #16545035 time: 2020-07-04T01:00:03 latency: 485 ms from: houdini-testnet irreversible: 16545027 (-8)
3486ms th_a application.cpp:499 handle_block ] Got block: #16545035 time: 2020-07-04T01:00:03 latency: 486 ms from: houdini-testnet irreversible: 16545027 (-8)
15038ms th_a application.cpp:499 handle_block ] Got block: #16545036 time: 2020-07-04T01:00:15 latency: 38 ms from: royal-flush irreversible: 16545027 (-9)
18022ms th_a application.cpp:499 handle_block ] Got block: #16545037 time: 2020-07-04T01:00:18 latency: 22 ms from: roelandp irreversible: 16545027 (-10)
21021ms th_a application.cpp:499 handle_block ] Got block: #16545038 time: 2020-07-04T01:00:21 latency: 21 ms from: eifos-witness irreversible: 16545027 (-11)
24023ms th_a application.cpp:499 handle_block ] Got block: #16545039 time: 2020-07-04T01:00:24 latency: 23 ms from: blckchnd-testnet irreversible: 16545027 (-12)
27185ms th_a application.cpp:499 handle_block ] Got block: #16545040 time: 2020-07-04T01:00:27 latency: 185 ms from: blockops1 irreversible: 16545027 (-13)
33162ms th_a application.cpp:499 handle_block ] Got block: #16545041 time: 2020-07-04T01:00:33 latency: 162 ms from: spacecrypt-witness irreversible: 16545027 (-14)
36023ms th_a application.cpp:499 handle_block ] Got block: #16545042 time: 2020-07-04T01:00:36 latency: 23 ms from: alexpu-witness irreversible: 16545027 (-15)
38988ms th_a application.cpp:499 handle_block ] Got block: #16545043 time: 2020-07-04T01:00:39 latency: -11 ms from: taconator-witness irreversible: 16545027 (-16)
45050ms th_a application.cpp:499 handle_block ] Got block: #16545044 time: 2020-07-04T01:00:45 latency: 50 ms from: houdini-testnet irreversible: 16545027 (-17)
48037ms th_a application.cpp:499 handle_block ] Got block: #16545045 time: 2020-07-04T01:00:48 latency: 37 ms from: royal-flush irreversible: 16545027 (-18)
53577ms th_a application.cpp:549 handle_transaction ] Got 1 transactions from network
1790415ms th_a application.cpp:637 get_item ] Couldn't find block 0082cafa85f03a85900d54a74000000000000000 -- corresponding ID in our chain is 0082cafa85f03a85900d54a72a5e560139f766f2```