Block 6130730 sync failure from v1.116.0 !2968 'Refactor memo'
When syncing a fullnode, I encountered a LastResultsHash conflict sync failure.
panic: Failed to process committed block (11292270:2ACA2B23EABD32300434B94037578A9EB730CC8DA3D3458944C42D216CB62BDF): wrong Block.Header.LastResultsHash. Expected F6DE4CE64DD2067DA88059A1667FAC7652BC5161E3A7FAE37D3BC0F3BABF8B24, got 7371C215F7006EAD21901D380145F6CF019CEEDDEEC025DEDFCF44B89CBE1CAB
Coming back to it now, block 11292269
( https://midgard.ninerealms.com/v2/debug/block/11292269 )
contains a "refund" event with reason "ticker error: too many characters".
Testing by syncing a fullnode (from a local archive fullnode) from post-hard-fork genesis (block 4786560) while running v1.119.0 code:
panic: Failed to process committed block (6130731:B03A689B95A97772B6F09A02694C3A6A88D632641207FA79502B9CF0462024FA): wrong Block.Header.LastResultsHash. Expected 3F290DC4C222CCC545CAB1FBDA502B31E7BD137CE3733E972E5784183110D6CF, got 108425BF4ED5256593E574C31036DB5F25218D34DBF7A4336B63FD10A97F3259
The network version at the time block 6130730 was originally produced was v1.91.0 .
https://thornode-v1.ninerealms.com/thorchain/version?height=6130730
(Block 11292269 was in v1.113.0 .)
Comparing from ten blocks earlier (data folder hypothetically available on request) with
thornode start --halt-height 6130730
,
localhost:1317/blocks/6130730
was identical (unchanged CRC-32),
but
localhost:27147/block_results?height=6130730
had notable differences;
in the diff, other than emitted event differences which are expected to not affect the LastResultsHash,
something that that stood out was (for one of the txs_reuslts) this:
< "gas_used": "18673105",
---
> "gas_used": "18675105",2023-09-07_v1.91.0_6130730_block_results__CRC32_F32C1A1D_.json
2023-09-07_v1.119.0_6130730_block_results__CRC32_F6B14A56_.json
2023-09-07_diff_between_v1.91.0_and_v1.119.0_6130730_block_results.txt
More specifically, comparing sequential commits c80298df and 8b55a43d (!2968 (merged) 'Refactor memo') from within v1.116.0 produced a smaller diff with the same gas_used change.
21005c21005
< "log": "[{\"events\":[{\"type\":\"coin_received\",\"attributes\":[{\"key\":\"receiver\",\"value\":\"thor1dheycdevq39qlkxs2a6wuuzyn4aqxhve4qxtxt\"},{\"key\":\"amount\",\"value\":\"2441726rune\"}]},{\"type\":\"coin_spent\",\"attributes\":[{\"key\":\"spender\",\"value\":\"thor1g98cy3n9mmjrpn0sxmn63lztelera37n8n67c0\"},{\"key\":\"amount\",\"value\":\"2441726rune\"}]},{\"type\":\"fee\",\"attributes\":[{\"key\":\"tx_id\",\"value\":\"56883027F10C26D3DD84FC162D77002ECFDD3B6C16B0565815AD0B3AFF7D1B25\"},{\"key\":\"coins\",\"value\":\"5133491 BNB.BUSD-BD1\"},{\"key\":\"pool_deduct\",\"value\":\"2441726\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"set_observed_txin\"},{\"key\":\"sender\",\"value\":\"thor1g98cy3n9mmjrpn0sxmn63lztelera37n8n67c0\"}]},{\"type\":\"refund\",\"attributes\":[{\"key\":\"code\",\"value\":\"105\"},{\"key\":\"reason\",\"value\":\"ticker error: too many characters\"},{\"key\":\"id\",\"value\":\"56883027F10C26D3DD84FC162D77002ECFDD3B6C16B0565815AD0B3AFF7D1B25\"},{\"key\":\"chain\",\"value\":\"BNB\"},{\"key\":\"from\",\"value\":\"bnb1jy7h4muz0ruflmrlxnt89ayhn3mf82sgza4vfm\"},{\"key\":\"to\",\"value\":\"bnb1mtc0p4s56sw49crr874ncuhatwny9gfyvrgf4m\"},{\"key\":\"coin\",\"value\":\"20000000000 BNB.BUSD-BD1\"},{\"key\":\"memo\",\"value\":\"=:THOR.thor1wztmenpsv5wn80ev6zzsmu5647gdrc0srgdfsz:9383480111:t.THOR:30\"}]},{\"type\":\"scheduled_outbound\",\"attributes\":[{\"key\":\"chain\",\"value\":\"BNB\"},{\"key\":\"to_address\",\"value\":\"bnb1jy7h4muz0ruflmrlxnt89ayhn3mf82sgza4vfm\"},{\"key\":\"vault_pub_key\",\"value\":\"thorpub1addwnpepq2mnukdcyduxr5wlzcn08fug6gtrlnz70w89d0sxt2lxapkcuksn6ec9gkq\"},{\"key\":\"coin_asset\",\"value\":\"BNB.BUSD-BD1\"},{\"key\":\"coin_amount\",\"value\":\"19994866509\"},{\"key\":\"coin_decimals\",\"value\":\"0\"},{\"key\":\"memo\",\"value\":\"REFUND:56883027F10C26D3DD84FC162D77002ECFDD3B6C16B0565815AD0B3AFF7D1B25\"},{\"key\":\"gas_rate\",\"value\":\"11250\"},{\"key\":\"in_hash\",\"value\":\"56883027F10C26D3DD84FC162D77002ECFDD3B6C16B0565815AD0B3AFF7D1B25\"},{\"key\":\"out_hash\"},{\"key\":\"module_name\"},{\"key\":\"max_gas_asset_0\",\"value\":\"BNB.BNB\"},{\"key\":\"max_gas_amount_0\",\"value\":\"7500\"},{\"key\":\"max_gas_decimals_0\",\"value\":\"8\"}]},{\"type\":\"transfer\",\"attributes\":[{\"key\":\"recipient\",\"value\":\"thor1dheycdevq39qlkxs2a6wuuzyn4aqxhve4qxtxt\"},{\"key\":\"sender\",\"value\":\"thor1g98cy3n9mmjrpn0sxmn63lztelera37n8n67c0\"},{\"key\":\"amount\",\"value\":\"2441726rune\"}]}]},{\"msg_index\":1,\"events\":[{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"set_observed_txout\"}]},{\"type\":\"outbound\",\"attributes\":[{\"key\":\"in_tx_id\",\"value\":\"C079C62BE73CFD128D4B599022B6E0E21E7401026ACA748EDA076AFFA93CBCFF\"},{\"key\":\"id\",\"value\":\"7ED8693C4A5D5491544F20EFD33DA931FFDD9C6F478BB5CD292C99D0358223D1\"},{\"key\":\"chain\",\"value\":\"BNB\"},{\"key\":\"from\",\"value\":\"bnb1npyeaz47m9yzu9s4ap8qf0y0tdpy8ypgqmf0yz\"},{\"key\":\"to\",\"value\":\"bnb1rv89nkw2x5ksvhf6jtqwqpke4qhh7jmudpvqmj\"},{\"key\":\"coin\",\"value\":\"502602263236 BNB.BUSD-BD1\"},{\"key\":\"memo\",\"value\":\"OUT:C079C62BE73CFD128D4B599022B6E0E21E7401026ACA748EDA076AFFA93CBCFF\"},{\"key\":\"in_tx_id\",\"value\":\"6C31DB026FA2D84CFF563138ADD32168DCE2A3F818F7BDA5F7EF968836136BDA\"},{\"key\":\"id\",\"value\":\"7C5493CC92C8150A8A8B2EE4A31DD2C4358628CFBF0DBCBF972E2C5E576C662B\"},{\"key\":\"chain\",\"value\":\"BNB\"},{\"key\":\"from\",\"value\":\"bnb1mtc0p4s56sw49crr874ncuhatwny9gfyvrgf4m\"},{\"key\":\"to\",\"value\":\"bnb145agzsdf7gl8lq94x2e7s9lyaul0dcqdtgc8ct\"},{\"key\":\"coin\",\"value\":\"83806413157 BNB.BUSD-BD1\"},{\"key\":\"memo\",\"value\":\"OUT:6C31DB026FA2D84CFF563138ADD32168DCE2A3F818F7BDA5F7EF968836136BDA\"}]},{\"type\":\"tss_keysign\",\"attributes\":[{\"key\":\"txid\",\"value\":\"7C5493CC92C8150A8A8B2EE4A31DD2C4358628CFBF0DBCBF972E2C5E576C662B\"},{\"key\":\"median_duration_ms\",\"value\":\"8127\"}]}]}]",
---
> "log": "[{\"events\":[{\"type\":\"coin_received\",\"attributes\":[{\"key\":\"receiver\",\"value\":\"thor1dheycdevq39qlkxs2a6wuuzyn4aqxhve4qxtxt\"},{\"key\":\"amount\",\"value\":\"2441726rune\"}]},{\"type\":\"coin_spent\",\"attributes\":[{\"key\":\"spender\",\"value\":\"thor1g98cy3n9mmjrpn0sxmn63lztelera37n8n67c0\"},{\"key\":\"amount\",\"value\":\"2441726rune\"}]},{\"type\":\"fee\",\"attributes\":[{\"key\":\"tx_id\",\"value\":\"56883027F10C26D3DD84FC162D77002ECFDD3B6C16B0565815AD0B3AFF7D1B25\"},{\"key\":\"coins\",\"value\":\"5133491 BNB.BUSD-BD1\"},{\"key\":\"pool_deduct\",\"value\":\"2441726\"}]},{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"set_observed_txin\"},{\"key\":\"sender\",\"value\":\"thor1g98cy3n9mmjrpn0sxmn63lztelera37n8n67c0\"}]},{\"type\":\"refund\",\"attributes\":[{\"key\":\"code\",\"value\":\"105\"},{\"key\":\"reason\",\"value\":\"9383480111 is not recognizable\"},{\"key\":\"id\",\"value\":\"56883027F10C26D3DD84FC162D77002ECFDD3B6C16B0565815AD0B3AFF7D1B25\"},{\"key\":\"chain\",\"value\":\"BNB\"},{\"key\":\"from\",\"value\":\"bnb1jy7h4muz0ruflmrlxnt89ayhn3mf82sgza4vfm\"},{\"key\":\"to\",\"value\":\"bnb1mtc0p4s56sw49crr874ncuhatwny9gfyvrgf4m\"},{\"key\":\"coin\",\"value\":\"20000000000 BNB.BUSD-BD1\"},{\"key\":\"memo\",\"value\":\"=:THOR.thor1wztmenpsv5wn80ev6zzsmu5647gdrc0srgdfsz:9383480111:t.THOR:30\"}]},{\"type\":\"scheduled_outbound\",\"attributes\":[{\"key\":\"chain\",\"value\":\"BNB\"},{\"key\":\"to_address\",\"value\":\"bnb1jy7h4muz0ruflmrlxnt89ayhn3mf82sgza4vfm\"},{\"key\":\"vault_pub_key\",\"value\":\"thorpub1addwnpepq2mnukdcyduxr5wlzcn08fug6gtrlnz70w89d0sxt2lxapkcuksn6ec9gkq\"},{\"key\":\"coin_asset\",\"value\":\"BNB.BUSD-BD1\"},{\"key\":\"coin_amount\",\"value\":\"19994866509\"},{\"key\":\"coin_decimals\",\"value\":\"0\"},{\"key\":\"memo\",\"value\":\"REFUND:56883027F10C26D3DD84FC162D77002ECFDD3B6C16B0565815AD0B3AFF7D1B25\"},{\"key\":\"gas_rate\",\"value\":\"11250\"},{\"key\":\"in_hash\",\"value\":\"56883027F10C26D3DD84FC162D77002ECFDD3B6C16B0565815AD0B3AFF7D1B25\"},{\"key\":\"out_hash\"},{\"key\":\"module_name\"},{\"key\":\"max_gas_asset_0\",\"value\":\"BNB.BNB\"},{\"key\":\"max_gas_amount_0\",\"value\":\"7500\"},{\"key\":\"max_gas_decimals_0\",\"value\":\"8\"}]},{\"type\":\"transfer\",\"attributes\":[{\"key\":\"recipient\",\"value\":\"thor1dheycdevq39qlkxs2a6wuuzyn4aqxhve4qxtxt\"},{\"key\":\"sender\",\"value\":\"thor1g98cy3n9mmjrpn0sxmn63lztelera37n8n67c0\"},{\"key\":\"amount\",\"value\":\"2441726rune\"}]}]},{\"msg_index\":1,\"events\":[{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"set_observed_txout\"}]},{\"type\":\"outbound\",\"attributes\":[{\"key\":\"in_tx_id\",\"value\":\"C079C62BE73CFD128D4B599022B6E0E21E7401026ACA748EDA076AFFA93CBCFF\"},{\"key\":\"id\",\"value\":\"7ED8693C4A5D5491544F20EFD33DA931FFDD9C6F478BB5CD292C99D0358223D1\"},{\"key\":\"chain\",\"value\":\"BNB\"},{\"key\":\"from\",\"value\":\"bnb1npyeaz47m9yzu9s4ap8qf0y0tdpy8ypgqmf0yz\"},{\"key\":\"to\",\"value\":\"bnb1rv89nkw2x5ksvhf6jtqwqpke4qhh7jmudpvqmj\"},{\"key\":\"coin\",\"value\":\"502602263236 BNB.BUSD-BD1\"},{\"key\":\"memo\",\"value\":\"OUT:C079C62BE73CFD128D4B599022B6E0E21E7401026ACA748EDA076AFFA93CBCFF\"},{\"key\":\"in_tx_id\",\"value\":\"6C31DB026FA2D84CFF563138ADD32168DCE2A3F818F7BDA5F7EF968836136BDA\"},{\"key\":\"id\",\"value\":\"7C5493CC92C8150A8A8B2EE4A31DD2C4358628CFBF0DBCBF972E2C5E576C662B\"},{\"key\":\"chain\",\"value\":\"BNB\"},{\"key\":\"from\",\"value\":\"bnb1mtc0p4s56sw49crr874ncuhatwny9gfyvrgf4m\"},{\"key\":\"to\",\"value\":\"bnb145agzsdf7gl8lq94x2e7s9lyaul0dcqdtgc8ct\"},{\"key\":\"coin\",\"value\":\"83806413157 BNB.BUSD-BD1\"},{\"key\":\"memo\",\"value\":\"OUT:6C31DB026FA2D84CFF563138ADD32168DCE2A3F818F7BDA5F7EF968836136BDA\"}]},{\"type\":\"tss_keysign\",\"attributes\":[{\"key\":\"txid\",\"value\":\"7C5493CC92C8150A8A8B2EE4A31DD2C4358628CFBF0DBCBF972E2C5E576C662B\"},{\"key\":\"median_duration_ms\",\"value\":\"8127\"}]}]}]",
21008c21008
< "gas_used": "18673105",
---
> "gas_used": "18675105",
21215c21215
< "value": "dGlja2VyIGVycm9yOiB0b28gbWFueSBjaGFyYWN0ZXJz",
---
> "value": "OTM4MzQ4MDExMSBpcyBub3QgcmVjb2duaXphYmxl",2023-09-08_c80298df_6130730_block_results__CRC32_F32C1A1D_.json
2023-09-08_8b55a43d_6130730_block_results__CRC32_4E604276_.json
2023-09-08_diff_between_c80298df_and_8b55a43d_6130730_block_results.txt
Translating from base64, in block 6130730
( https://midgard.ninerealms.com/v2/debug/block/6130730 )
this was a gas_used change associated with a change in refund reason "ticker error: too many characters"
( https://gitlab.com/thorchain/thornode/-/blob/c80298df/common/ticker.go#L32 )
(from NewTicker)
(same as in block 11292269)
to "9383480111 is not recognizable"
( https://gitlab.com/thorchain/thornode/-/blob/8b55a43d/x/thorchain/memo/memo.go#L265 )
(from FetchAddress)
. The txs_results transaction's memo:
"=:THOR.thor1wztmenpsv5wn80ev6zzsmu5647gdrc0srgdfsz:9383480111:t.THOR:30"
This was similar to the memo in block 11292269:
"=:ETH.0xdAC17F958D2ee523a2206206994597C13D831ec7:0xAC6533c9F1e642C38B4c5ed35A7d76f508F7c039:3650171013:tr:0"
.
Code paths:
https://gitlab.com/thorchain/thornode/-/blob/c80298df/x/thorchain/handler_observed_txin.go#L208
memo, _ := ParseMemoWithTHORNames(ctx, h.mgr.Keeper(), tx.Tx.Memo) // ignore err
https://gitlab.com/thorchain/thornode/-/blob/c80298df/x/thorchain/memo/memo.go#L307
mem, parts, err := parseBase(keeper.GetVersion(), memo)
https://gitlab.com/thorchain/thornode/-/blob/c80298df/x/thorchain/memo/memo.go#L230
mem.Asset, err = common.NewAssetWithShortCodes(version, parts[1])
https://gitlab.com/thorchain/thornode/-/blob/c80298df/common/asset.go#L108
return NewAsset(input)
https://gitlab.com/thorchain/thornode/-/blob/c80298df/common/asset.go#L78
asset.Ticker, err = NewTicker(parts[0])
https://gitlab.com/thorchain/thornode/-/blob/c80298df/common/ticker.go#L32
return noTicker, errors.New("ticker error: too many characters")
https://gitlab.com/thorchain/thornode/-/blob/8b55a43d/x/thorchain/handler_observed_txin.go#L208
memo, _ := ParseMemoWithTHORNames(ctx, h.mgr.Keeper(), tx.Tx.Memo) // ignore err
https://gitlab.com/thorchain/thornode/-/blob/8b55a43d/x/thorchain/memo/memo.go#L239
return parser.parse()
https://gitlab.com/thorchain/thornode/-/blob/8b55a43d/x/thorchain/memo/memo_parser.go#L79
return p.ParseSwapMemo()
https://gitlab.com/thorchain/thornode/-/blob/8b55a43d/x/thorchain/memo/memo_swap.go#L125
return ParseSwapMemoV1(p.ctx, p.keeper, p.getAsset(1, true, common.EmptyAsset), p.parts)
https://gitlab.com/thorchain/thornode/-/blob/8b55a43d/x/thorchain/memo/memo_swap_archive.go#L257
destination, err = FetchAddress(ctx, keeper, parts[2], asset.Chain)
https://gitlab.com/thorchain/thornode/-/blob/8b55a43d/x/thorchain/memo/memo.go#L257
if keeper.THORNameExists(ctx, parts[0]) {
(This KVStore check, returning false, might be the gas_used difference source.)
https://gitlab.com/thorchain/thornode/-/blob/8b55a43d/x/thorchain/memo/memo.go#L265
return common.NoAddress, fmt.Errorf("%s is not recognizable", name)
My initial code proposal (suggestions welcome; one commit added to v1.119.0),
able to sync through 6130730 without failure (later blocks now being checked):
da775cff7
At time of writing this has synced from block 6130721 to 6134293.
2023-09-08_da775cff7_6130730_block_results__CRC32_FB175ECF_.json
Having been rebased onto develop, similarly able to sync through 6130730 (identical block_results CRC-32):
711dd63d
Namely, !3116 (merged)
'Resolve block 6130730 sync failure'.
Update:
I am for example curious about whether there will be a related sync failure in blocks 9844766
(through 9844826) for withdraws,
warranting a similar resolution, but it will take me a while to sync to there.
|
Further update:
It now appears to me that a commit will not be needed for the withdraw-memo's same refund reason,
as I believe ParseWithdrawLiquidityMemoV1 should here reach
withdrawalAsset, err = common.NewAsset(parts[3])
(and return the same error) without a KVStore check difference
(particularly as withdraw memo parsing unlike swap memo parsing does not check for addresses, only pool asset and withdrawal asset).
I could of course be mistaken, but I do not currently see reason to pre-emptively add a further commit.
2023-09-24 update:
Pleasingly (at time of writing) the synchronisation has passed through {blocks 9844766 through 9844826} indeed without sync failure, at time of writing reaching block 9863447 (and continuing to sync).
The "ticker error: too many characters" refund reason was also not altered for block 9844766, due to the similar NewAsset error return from here.