Ungraceful shutdown when interrupting software with CTRL+C
This problem is visible on develop branch, from commit 1846b170
After fixing problems with database corruption and witness scheduler, a new problem pops up, that was apparently hidden by previous two, as they were happening first in a row of errors.
When software is interrupted with CTRL+C during synchronization phase, software ungracefully shutdowns, with a huge error log. Note, that this looks more like cosmetic problem, than functional one. Regardless of it, we are able to continue replay/syncing close to the point where the software is interrupted.
The root cause appears to be in a lack of thread synchronization, and bad handling of CTRL+C signals. Looks like many threads are not aware that software is shutting down, and they continue to execute while the rest of the software is shutting down.
Error logs (actual outputs are much larger):
1989174ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T16:00:00
1989237ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
1989549ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T17:00:00
1989611ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
1989922ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T18:00:00
1989984ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
1990261ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T19:00:00
1990274ms asio main.cpp:168 operator() ] Caught SIGINT attempting to exit cleanly
1990323ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
1990417ms th_a main.cpp:181 main ] Exiting from signal 2
1990424ms th_a application.cpp:460 handle_block ] Got block: #685294 time: 2017-06-30T19:22:00 latency: 155175070424 ms from: scissors-witness irreversible: 685262 (-32)
1990425ms th_a application.cpp:460 handle_block ] Got block: #685295 time: 2017-06-30T19:22:03 latency: 155175067425 ms from: billbutler-witness irreversible: 685263 (-32)
1990425ms th_a application.cpp:460 handle_block ] Got block: #685296 time: 2017-06-30T19:22:06 latency: 155175064425 ms from: pfunk-witness irreversible: 685264 (-32)
1990426ms th_a application.cpp:460 handle_block ] Got block: #685297 time: 2017-06-30T19:22:09 latency: 155175061425 ms from: joseph-witness irreversible: 685265 (-32)
1990426ms th_a application.cpp:460 handle_block ] Got block: #685298 time: 2017-06-30T19:22:12 latency: 155175058426 ms from: content-junkie irreversible: 685266 (-32)
1990426ms th_a application.cpp:460 handle_block ] Got block: #685299 time: 2017-06-30T19:22:15 latency: 155175055426 ms from: ausbitbank-witness irreversible: 685267 (-32)
1990427ms th_a application.cpp:460 handle_block ] Got block: #685300 time: 2017-06-30T19:22:18 latency: 155175052427 ms from: bhuz irreversible: 685268 (-32)
1990427ms th_a application.cpp:460 handle_block ] Got block: #685301 time: 2017-06-30T19:22:21 latency: 155175049427 ms from: xeldal-witness irreversible: 685269 (-32)
1990428ms th_a application.cpp:460 handle_block ] Got block: #685302 time: 2017-06-30T19:22:24 latency: 155175046428 ms from: thecryptodrive-witness irreversible: 685270 (-32)
1990428ms th_a application.cpp:460 handle_block ] Got block: #685303 time: 2017-06-30T19:22:27 latency: 155175043428 ms from: winner.winner.chicken.dinner irreversible: 685271 (-32)
1990428ms th_a application.cpp:460 handle_block ] Got block: #685304 time: 2017-06-30T19:22:30 latency: 155175040428 ms from: nuevax irreversible: 685272 (-32)
1990428ms th_a application.cpp:460 handle_block ] Got block: #685305 time: 2017-06-30T19:22:33 latency: 155175037428 ms from: techadept-witness irreversible: 685273 (-32)
1990429ms th_a application.cpp:460 handle_block ] Got block: #685306 time: 2017-06-30T19:22:36 latency: 155175034429 ms from: pfunk-witness irreversible: 685274 (-32)
1990429ms th_a application.cpp:460 handle_block ] Got block: #685307 time: 2017-06-30T19:22:39 latency: 155175031429 ms from: tao-witness irreversible: 685274 (-33)
1990429ms th_a application.cpp:460 handle_block ] Got block: #685308 time: 2017-06-30T19:22:42 latency: 155175028429 ms from: anyx irreversible: 685274 (-34)
1990430ms th_a application.cpp:460 handle_block ] Got block: #685309 time: 2017-06-30T19:22:45 latency: 155175025430 ms from: taconator-witness irreversible: 685275 (-34)
1990430ms th_a application.cpp:460 handle_block ] Got block: #685310 time: 2017-06-30T19:22:48 latency: 155175022430 ms from: baxters-sports-witness irreversible: 685275 (-35)
1990431ms th_a application.cpp:460 handle_block ] Got block: #685311 time: 2017-06-30T19:22:51 latency: 155175019431 ms from: melea-trust irreversible: 685275 (-36)
1990431ms th_a application.cpp:460 handle_block ] Got block: #685312 time: 2017-06-30T19:22:54 latency: 155175016431 ms from: bhuz irreversible: 685275 (-37)
1990431ms th_a application.cpp:460 handle_block ] Got block: #685313 time: 2017-06-30T19:22:57 latency: 155175013431 ms from: firecrab-witness irreversible: 685275 (-38)
1990432ms th_a application.cpp:460 handle_block ] Got block: #685314 time: 2017-06-30T19:23:00 latency: 155175010432 ms from: bts-genx4217 irreversible: 685277 (-37)
1990432ms th_a application.cpp:460 handle_block ] Got block: #685315 time: 2017-06-30T19:23:03 latency: 155175007432 ms from: roelandp irreversible: 685278 (-37)
1990433ms th_a application.cpp:460 handle_block ] Got block: #685316 time: 2017-06-30T19:23:06 latency: 155175004433 ms from: winner.winner.chicken.dinner irreversible: 685279 (-37)
1990433ms th_a application.cpp:460 handle_block ] Got block: #685317 time: 2017-06-30T19:23:09 latency: 155175001433 ms from: jesta irreversible: 685279 (-38)
1990434ms th_a application.cpp:460 handle_block ] Got block: #685318 time: 2017-06-30T19:23:12 latency: 155174998434 ms from: techadept-witness irreversible: 685279 (-39)
1990435ms th_a application.cpp:460 handle_block ] Got block: #685319 time: 2017-06-30T19:23:15 latency: 155174995434 ms from: riverhead irreversible: 685279 (-40)
1990435ms th_a application.cpp:460 handle_block ] Got block: #685320 time: 2017-06-30T19:23:18 latency: 155174992435 ms from: bacchist-witness irreversible: 685280 (-40)
1990435ms th_a application.cpp:460 handle_block ] Got block: #685321 time: 2017-06-30T19:23:21 latency: 155174989435 ms from: jburr-witness irreversible: 685281 (-40)
1990436ms th_a application.cpp:460 handle_block ] Got block: #685322 time: 2017-06-30T19:23:24 latency: 155174986436 ms from: cryptofreedom-witness irreversible: 685281 (-41)
1990436ms th_a application.cpp:460 handle_block ] Got block: #685323 time: 2017-06-30T19:23:27 latency: 155174983436 ms from: xtar-testz irreversible: 685282 (-41)
1990437ms th_a application.cpp:460 handle_block ] Got block: #1 time: 2017-06-30T19:23:30 latency: 155174980437 ms from: judge-dredd irreversible: 685282 (-4294282015)
1990437ms th_a application.cpp:491 handle_block ] Error when pushing block:
10 assert_exception: Assert Exception
item->num > std::max<int64_t>( 0, int64_t(_head->num) - (_max_size) ): attempting to push a block that is too old
{"item->num":1,"head":685323,"max_size":42}
th_a fork_database.cpp:77 _push_block
{"new_block":{"previous":"00000000000000000000000000000000105f82d3","timestamp":"2017-06-30T19:23:30","witness":"1.6.58","next_secret_hash":"6b0103deea9e344d8f01da4b486a4b55cc9b370d","previous_secret":"4ff4dbaa769bda49c72546d9b20e2791c35c19bd","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f1ff07820ce032190310df2d52ed63c8fbf98552b96c34fbc66303fbfcdf645d03ce4b829747adccc2fdd8410b037934fca6c4336d77fcf800ba0042d80b04727","transactions":[]}}
th_a db_block.cpp:293 _push_block
1990437ms th_a application.cpp:460 handle_block ] Got block: #2156031612 time: 2017-06-30T19:23:33 latency: 155174977437 ms from: good-karma irreversible: 685282 (-2155346330)
1990437ms th_a fork_database.cpp:65 push_block ] Pushing block to fork database that failed to link: 80826e7cfbfe2ebe91db7e024b12fbe0cfaa0212, 2156031612
1990437ms th_a fork_database.cpp:66 push_block ] Head: 685323, 000a750b5dcbf7d08034708824789556105f82d3
1990438ms th_a application.cpp:488 handle_block ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
{}
th_a fork_database.cpp:86 _push_block
{"new_block":{"previous":"80826e7b857f0000a0510a35857f0000b710c899","timestamp":"2017-06-30T19:23:33","witness":"1.6.37","next_secret_hash":"2b68b4819bb7c661754d17594ad66b8ccc0b2a17","previous_secret":"e2f0c4c8974574af28a7020da88b67a136c5cc89","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f5470911c62eb4264111204b8611a868d55016d05037199d1016c1983ce1d8e8b735b5b1c0552b638898a5fdb1fecfe251207e1ae5996d569a8c2de956c0cd895","transactions":[]}}
th_a db_block.cpp:293 _push_block
1990462ms th_a application.cpp:460 handle_block ] Got block: #685405 time: 2017-06-30T19:27:33 latency: 155174737462 ms from: someguy123 irreversible: 685282 (-123)
1990463ms th_a fork_database.cpp:65 push_block ] Pushing block to fork database that failed to link: 000a755d7e9bbd34fce2824e947184cd7aa49843, 685405
1990463ms th_a fork_database.cpp:66 push_block ] Head: 685323, 000a750b5dcbf7d08034708824789556105f82d3
1990463ms th_a application.cpp:488 handle_block ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
{}
th_a fork_database.cpp:86 _push_block
{"new_block":{"previous":"000a755cf217da12c0466ff0db71a62c26e4d487","timestamp":"2017-06-30T19:27:33","witness":"1.6.24","next_secret_hash":"5912def6602ae21dc8726aa10675cf78f562923c","previous_secret":"7da054df4265eb8a60f22b034f9a7537e69b15ba","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"20787a1f99ceb35e2843f81aa8964890d4503bab3aa9b4a68dd082640b55856fcf5f4c887c4880b2123fa9310f5707463327035fa0d02f3a43172b601b0c7f872e","transactions":[]}}
th_a db_block.cpp:293 _push_block
1990463ms th_a application.cpp:460 handle_block ] Got block: #685406 time: 2017-06-30T19:27:36 latency: 155174734463 ms from: billbutler-witness irreversible: 685282 (-124)
1990463ms th_a fork_database.cpp:65 push_block ] Pushing block to fork database that failed to link: 000a755e27c60759bb32a07eafc3d4fbca276f0e, 685406
1990463ms th_a fork_database.cpp:66 push_block ] Head: 685323, 000a750b5dcbf7d08034708824789556105f82d3
1990463ms th_a application.cpp:488 handle_block ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
{}
th_a fork_database.cpp:86 _push_block
{"new_block":{"previous":"000a755d7e9bbd34fce2824e947184cd7aa49843","timestamp":"2017-06-30T19:27:36","witness":"1.6.18","next_secret_hash":"a8dcf797cb13ba6279ddd306e806136e9f2402d2","previous_secret":"1a2c2efa35382f91d86eadf81c0139d02935bfd4","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f5513fad376f2c917415adedb469848a23b8c6fefc32630a37c37e1fafac46f703a9473f45d6b41f9c3127c3e383338a505e494e4146158474cfc29c5e0ca4ff0","transactions":[]}}
th_a db_block.cpp:293 _push_block
witness_node: /peerplays/libraries/net/node.cpp:2765: void graphene::net::detail::node_impl::on_blockchain_item_ids_inventory_message(graphene::net::peer_connection*, const graphene::net::blockchain_item_ids_inventory_message&): Assertion `false && "exceptions not expected here"' failed.
Aborted (core dumped)2586468ms th_a application.cpp:460 handle_block ] Got block: #1120000 time: 2017-07-15T23:29:12 latency: 153864834468 ms from: riverhead irreversible: 1119968 (-32)
2586598ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-07-16T00:00:00
2586660ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2586923ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-07-16T01:00:00
2586985ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2587257ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-07-16T02:00:00
2587318ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2587579ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-07-16T03:00:00
2587641ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2587928ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-07-16T04:00:00
2587991ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2588255ms asio main.cpp:168 operator() ] Caught SIGINT attempting to exit cleanly
2588294ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-07-16T05:00:00
2588356ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2588410ms th_a main.cpp:181 main ] Exiting from signal 2
2588422ms th_a application.cpp:460 handle_block ] Got block: #1126865 time: 2017-07-16T05:13:42 latency: 153844166422 ms from: hiltos-witness irreversible: 1126825 (-40)
2588422ms th_a db_block.cpp:287 _push_block ] Failed to push new block:
10 assert_exception: Assert Exception
next_block.witness == scheduled_witness: Witness produced block at wrong time
{"block witness":"1.6.28","scheduled":"1.6.39","slot_num":200}
th_a db_block.cpp:898 validate_block_header
{"next_block.block_num()":1127064}
th_a db_block.cpp:753 _apply_block
2588422ms th_a application.cpp:491 handle_block ] Error when pushing block:
10 assert_exception: Assert Exception
next_block.witness == scheduled_witness: Witness produced block at wrong time
{"block witness":"1.6.28","scheduled":"1.6.39","slot_num":200}
th_a db_block.cpp:898 validate_block_header
{"next_block.block_num()":1127064}
th_a db_block.cpp:753 _apply_block
{"new_block":{"previous":"00113297d72a20ad3ca02d2b6d427801693daab6","timestamp":"2017-07-16T05:23:39","witness":"1.6.28","next_secret_hash":"bff70b1ba2b596a39deb02ea8392cd52f0093a61","previous_secret":"f6a72c7ffb1a78a17735f4424d1e9e24011923b3","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"1f5f3c571eaf5274b4b0d3dad1315953e5f6ff5aa9d1016d9480e8bd7d16daf99600d00f943cb96811b39293af13352df3b5f23c7581bf82dd84287e4d052dd4e0","transactions":[]}}
th_a db_block.cpp:293 _push_block
2588422ms th_a db_block.cpp:219 _push_block ] Switching to fork: 001131d2190608e962526d8cca18bf6b49a1ceb5
2588422ms th_a db_block.cpp:225 _push_block ] popping block #1126864 001131d0ebfb21dde42d24b1d7affaf0872722f9
2588422ms th_a db_block.cpp:232 _push_block ] pushing block from fork #1126864 001131d0ebfb21dde42d24b1d7affaf0872722f9
2588422ms th_a db_block.cpp:232 _push_block ] pushing block from fork #1126865 001131d1f49716f921266ab2925b7263facba7c4
2588423ms th_a db_block.cpp:232 _push_block ] pushing block from fork #1126866 001131d2190608e962526d8cca18bf6b49a1ceb5
2588423ms th_a fork_database.cpp:65 push_block ] Pushing block to fork database that failed to link: 0011329aa1311900d55d31ff289567969d61d66d, 1127066
2588423ms th_a fork_database.cpp:66 push_block ] Head: 1126865, 001131d1f49716f921266ab2925b7263facba7c4
2588423ms th_a application.cpp:488 handle_block ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
{}
th_a fork_database.cpp:86 _push_block
{"new_block":{"previous":"00113299be854b27c4846a8f913f2172b7b9d6f3","timestamp":"2017-07-16T05:23:45","witness":"1.6.20","next_secret_hash":"ac3ec09255fa5ab0be7b90974730f845b756fabf","previous_secret":"e59ee1fe942b65c5ae44d0e787541a15206b9096","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"205f0b0057475b92bb0387aee43c811301e24bbeda56f7b49a7bf2cf0776026e5a1fad2857775ce0521b81f06d8069d831eb34cf92e642bde5a550b372be03bcb7","transactions":[]}}
th_a db_block.cpp:293 _push_block
2588423ms th_a fork_database.cpp:65 push_block ] Pushing block to fork database that failed to link: 0011329ef26f448bec7dc373805229768ef7ed7a, 1127070
2588423ms th_a fork_database.cpp:66 push_block ] Head: 1126865, 001131d1f49716f921266ab2925b7263facba7c4
2588423ms th_a application.cpp:488 handle_block ] Error when pushing block:
3080000 unlinkable_block_exception: unlinkable block
block does not link to known chain
{}
th_a fork_database.cpp:86 _push_block
{"new_block":{"previous":"0011329d9ce780ae9b23e2376a0d0dbc0e614830","timestamp":"2017-07-16T05:23:57","witness":"1.6.47","next_secret_hash":"56a0fcbe4b0585ca85d20ff52e826ce5edcfeb13","previous_secret":"fe6cd3e2d06830a31472bf4f986806ef156ca626","transaction_merkle_root":"0000000000000000000000000000000000000000","extensions":[],"witness_signature":"20557c2eb169b62b33dd36042615dce449c7247ab6b0dca641cad83b63330a62403f7440033786343a0eba02486d674d9777c2aab8c744dbc77fd92ea347e78697","transactions":[]}}
th_a db_block.cpp:293 _push_block- If interrupted during replay, everything runs just fine
ubuntu:/peerplays/peerplays-networks/mainnet$ ./witness_node
2420470ms th_a accounts_list_plugin.cpp:128 list_accounts ] accounts list plugin: list_accounts()
2420470ms th_a bookie_plugin.cpp:412 plugin_initialize ] bookie plugin: plugin_startup() begin
2420470ms th_a db_management.cpp:295 force_slow_replays ] enabling slow replays
2420471ms th_a bookie_plugin.cpp:435 plugin_initialize ] bookie plugin: plugin_startup() end
2420471ms th_a witness.cpp:91 plugin_initialize ] witness plugin: plugin_initialize() begin
2420471ms th_a witness.cpp:105 plugin_initialize ] Public Key: PPY6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV
2420471ms th_a witness.cpp:123 plugin_initialize ] witness plugin: plugin_initialize() end
2420471ms th_a object_database.cpp:106 open ] Opening object database from /peerplays/peerplays-networks/mainnet/witness_node_data_dir/blockchain ...
2420830ms th_a object_database.cpp:111 open ] Done opening object database.
2420831ms th_a db_management.cpp:109 reindex ] reindexing blockchain
2420831ms th_a db_management.cpp:114 reindex ] Replaying blocks, starting at 685283...
2420907ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T20:00:00
2420967ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2421086ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T21:00:00
2421147ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2421263ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T22:00:00
2421324ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2421439ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T23:00:00
2421500ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2421615ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-07-01T00:00:00
2421677ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2421790ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-07-01T01:00:00
2421851ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2421968ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-07-01T02:00:00
2422028ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2422148ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-07-01T03:00:00
2422209ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2422333ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-07-01T04:00:00
2422394ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
^C
ubuntu:/peerplays/peerplays-networks/mainnet$ ./witness_node
2422992ms th_a accounts_list_plugin.cpp:128 list_accounts ] accounts list plugin: list_accounts()
2422992ms th_a bookie_plugin.cpp:412 plugin_initialize ] bookie plugin: plugin_startup() begin
2422992ms th_a db_management.cpp:295 force_slow_replays ] enabling slow replays
2422992ms th_a bookie_plugin.cpp:435 plugin_initialize ] bookie plugin: plugin_startup() end
2422992ms th_a witness.cpp:91 plugin_initialize ] witness plugin: plugin_initialize() begin
2422992ms th_a witness.cpp:105 plugin_initialize ] Public Key: PPY6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV
2422992ms th_a witness.cpp:123 plugin_initialize ] witness plugin: plugin_initialize() end
2422992ms th_a object_database.cpp:106 open ] Opening object database from /peerplays/peerplays-networks/mainnet/witness_node_data_dir/blockchain ...
2423343ms th_a object_database.cpp:111 open ] Done opening object database.
2423343ms th_a db_management.cpp:109 reindex ] reindexing blockchain
2423344ms th_a db_management.cpp:114 reindex ] Replaying blocks, starting at 685283...
2423418ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T20:00:00
2423478ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2423590ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T21:00:00
2423650ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2423759ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T22:00:00
2423818ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
^C
ubuntu:/peerplays/peerplays-networks/mainnet$ ./witness_node
2424559ms th_a accounts_list_plugin.cpp:128 list_accounts ] accounts list plugin: list_accounts()
2424559ms th_a bookie_plugin.cpp:412 plugin_initialize ] bookie plugin: plugin_startup() begin
2424559ms th_a db_management.cpp:295 force_slow_replays ] enabling slow replays
2424559ms th_a bookie_plugin.cpp:435 plugin_initialize ] bookie plugin: plugin_startup() end
2424559ms th_a witness.cpp:91 plugin_initialize ] witness plugin: plugin_initialize() begin
2424559ms th_a witness.cpp:105 plugin_initialize ] Public Key: PPY6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV
2424559ms th_a witness.cpp:123 plugin_initialize ] witness plugin: plugin_initialize() end
2424559ms th_a object_database.cpp:106 open ] Opening object database from /peerplays/peerplays-networks/mainnet/witness_node_data_dir/blockchain ...
2424905ms th_a object_database.cpp:111 open ] Done opening object database.
2424905ms th_a db_management.cpp:109 reindex ] reindexing blockchain
2424905ms th_a db_management.cpp:114 reindex ] Replaying blocks, starting at 685283...
2424975ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T20:00:00
2425034ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2425143ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T21:00:00
2425202ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2425312ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T22:00:00
2425372ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2425480ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T23:00:00
2425539ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
^C
ubuntu:/peerplays/peerplays-networks/mainnet$ ./witness_node
2426008ms th_a accounts_list_plugin.cpp:128 list_accounts ] accounts list plugin: list_accounts()
2426008ms th_a bookie_plugin.cpp:412 plugin_initialize ] bookie plugin: plugin_startup() begin
2426008ms th_a db_management.cpp:295 force_slow_replays ] enabling slow replays
2426008ms th_a bookie_plugin.cpp:435 plugin_initialize ] bookie plugin: plugin_startup() end
2426008ms th_a witness.cpp:91 plugin_initialize ] witness plugin: plugin_initialize() begin
2426009ms th_a witness.cpp:105 plugin_initialize ] Public Key: PPY6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV
2426009ms th_a witness.cpp:123 plugin_initialize ] witness plugin: plugin_initialize() end
2426009ms th_a object_database.cpp:106 open ] Opening object database from /peerplays/peerplays-networks/mainnet/witness_node_data_dir/blockchain ...
2426375ms th_a object_database.cpp:111 open ] Done opening object database.
2426375ms th_a db_management.cpp:109 reindex ] reindexing blockchain
2426375ms th_a db_management.cpp:114 reindex ] Replaying blocks, starting at 685283...
2426451ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T20:00:00
2426511ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2426629ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T21:00:00
2426690ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2426805ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T22:00:00
2426866ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2426982ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-06-30T23:00:00
2427042ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
2427157ms th_a db_maint.cpp:1739 process_dividend_ass ] In process_dividend_assets time 2017-07-01T00:00:00
2427218ms th_a db_maint.cpp:763 update_active_sons ] Active SONs set NOT CHANGED
^C