tezos-node: crashed on “Inconsistent hash”
On April 1st 2020 around Block 890,840 (+/- 20 blocks), a node was seemingly rendered non-operational by looping on a inconsistent hash error. Worth noting that this only occurred on a single node out of a cluster of nodes.
There were no rejected blocks.
Node functionality was restored by removing .tezos-node and performing a snapshot import.
Any suggestions on why this may have happened and what to do in case this occurs again?
~/tezos$ ./tezos-admin-client list rejected blocks
No invalid blocks.
~/tezos$ ./tezos-client bootstrapped
Current head: BLp58HCYrrSc (timestamp: 2020-04-01T19:19:25-00:00, validation: 2020-04-01T21:12:02-00:00)
Apr 1 12:43:50 - validator.peer: Worker started for NetXdQprcVkpa:idqxBjD6pLM5
Apr 1 12:43:50 - validator.block: Validation of block BM6BDR8a1fBgCKufUTovXafy228m4igbdVqSeXeZq6cQBPwx9qQ failed
Apr 1 12:43:50 - validator.block: Request pushed on 2020-04-01T19:43:50-00:00, treated in 6.697us, completed in 247ms ,
Apr 1 12:43:50 - validator.block: Inconsistent hash:
Apr 1 12:43:50 - validator.block: got: CoUxJfUhEZSMt2CEs1s5T7mg2Z1kFPscJ4uM5mJvzCo74Cr6cHZB
Apr 1 12:43:50 - validator.block: expected: CoW5so6fDEurK5wv8ayoHTJ4VonEfbDoaU9sb6rpPLFEDuifhHMW
Apr 1 12:43:50 - node.validator.bootstrap_pipeline: Unexpected error (validator):
Apr 1 12:43:50 - node.validator.bootstrap_pipeline: Error:
Apr 1 12:43:50 - node.validator.bootstrap_pipeline: Inconsistent hash:
Apr 1 12:43:50 - node.validator.bootstrap_pipeline: got: CoUxJfUhEZSMt2CEs1s5T7mg2Z1kFPscJ4uM5mJvzCo74Cr6cHZB
Apr 1 12:43:50 - node.validator.bootstrap_pipeline: expected: CoW5so6fDEurK5wv8ayoHTJ4VonEfbDoaU9sb6rpPLFEDuifhHMW
Apr 1 12:43:50 - node.validator.bootstrap_pipeline:
Apr 1 12:43:50 - validator.peer: Worker crashed [validator-peer_37145]:
Apr 1 12:43:50 - validator.peer: Inconsistent hash:
Apr 1 12:43:50 - validator.peer: got: CoUxJfUhEZSMt2CEs1s5T7mg2Z1kFPscJ4uM5mJvzCo74Cr6cHZB
Apr 1 12:43:50 - validator.peer: expected: CoW5so6fDEurK5wv8ayoHTJ4VonEfbDoaU9sb6rpPLFEDuifhHMW
Apr 1 12:43:51 - validator.block: Validation of block BM6BDR8a1fBgCKufUTovXafy228m4igbdVqSeXeZq6cQBPwx9qQ failed
Apr 1 12:43:51 - validator.block: Request pushed on 2020-04-01T19:43:51-00:00, treated in 3.561us, completed in 252ms ,
Apr 1 12:43:51 - validator.block: Inconsistent hash:
Apr 1 12:43:51 - validator.block: got: CoUxJfUhEZSMt2CEs1s5T7mg2Z1kFPscJ4uM5mJvzCo74Cr6cHZB
Apr 1 12:43:51 - validator.block: expected: CoW5so6fDEurK5wv8ayoHTJ4VonEfbDoaU9sb6rpPLFEDuifhHMW
Cross-posted from https://tezos.stackexchange.com/questions/2380/how-to-handle-nodes-that-encounter-inconsistent-hash-failures