[Backwards-compatibility] Block 11783100 sync failure with v1.130.0-v1.124.0 (but not v1.123.0) THORNodes
Summary of the below:
From commit ef1d46a1, historical ParseMemo
calls (which have nil
for their p.keeper
)
panic at p.keeper.GetVersion()
(resulting in sync failure),
for which !3523 (merged) now proposes changing to p.version
instead.
V130 being a round number, I began a sync-from-genesis using it.
Rather than reaching a consensus failure, this appears to suddenly quit from a panic,
and attempting to resume the fullnode exits in the same way.
Relevant (183 blocks earlier):
11782917 (2023-07-20): YggFundLimit Mimir key (Constants 50 ->) 0 (triggering !2996 returns from all Yggdrasil vaults)
v1.130.0 fullnode logs:
(v1.130.0 logs collapsible section)
11:58AM INF github.com/tendermint/tendermint@v0.34.14/state/execution.go:235 > committed state app_hash=5A96A22E8A420DAA6ED8BB43CEBB87398EA964E86E79E49B1606FC1337189D36 height=11783099 module=state num_txs=0
11:58AM INF gitlab.com/thorchain/thornode/x/thorchain/manager_network_v115.go:697 > maximum rune deployed from POL
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x78 pc=0x2049d1d]
goroutine 450 [running]:
gitlab.com/thorchain/thornode/x/thorchain.(*TxOutStorageV116).addToBlockOut(_, {{0x3c76d90, 0xc00005a040}, {0x3c8a1e0, 0xc0f1d5d880}, {{0xb, 0x0}, {0xc1d7194510, 0x14}, 0xb3cbbc, ...}, ...}, ...)
gitlab.com/thorchain/thornode/x/thorchain/manager_txout_v116.go:680 +0x3fd
gitlab.com/thorchain/thornode/x/thorchain.(*TxOutStorageV116).UnSafeAddTxOutItem(_, {{0x3c76d90, 0xc00005a040}, {0x3c8a1e0, 0xc0f1d5d880}, {{0xb, 0x0}, {0xc1d7194510, 0x14}, 0xb3cbbc, ...}, ...}, ...)
gitlab.com/thorchain/thornode/x/thorchain/manager_txout_v116.go:237 +0x1b8
gitlab.com/thorchain/thornode/x/thorchain.(*ValidatorMgrV116).RequestYggReturn(_, {{0x3c76d90, 0xc00005a040}, {0x3c8a1e0, 0xc0f1d5d880}, {{0xb, 0x0}, {0xc1d7194510, 0x14}, 0xb3cbbc, ...}, ...}, ...)
gitlab.com/thorchain/thornode/x/thorchain/manager_validator_v116.go:1091 +0xe89
gitlab.com/thorchain/thornode/x/thorchain.(*ValidatorMgrV116).recallYggFunds(_, {{0x3c76d90, 0xc00005a040}, {0x3c8a1e0, 0xc0f1d5d880}, {{0xb, 0x0}, {0xc1d7194510, 0x14}, 0xb3cbbc, ...}, ...}, ...)
gitlab.com/thorchain/thornode/x/thorchain/manager_validator_v116.go:1124 +0x7f0
gitlab.com/thorchain/thornode/x/thorchain.(*ValidatorMgrV116).EndBlock(_, {{0x3c76d90, 0xc00005a040}, {0x3c8a1e0, 0xc0f1d5d880}, {{0xb, 0x0}, {0xc1d7194510, 0x14}, 0xb3cbbc, ...}, ...}, ...)
gitlab.com/thorchain/thornode/x/thorchain/manager_validator_v116.go:272 +0x390
gitlab.com/thorchain/thornode/x/thorchain.AppModule.EndBlock({{}, _, {{_, _}, {_, _}, {_, _}}, _}, {{0x3c76d90, ...}, ...}, ...)
gitlab.com/thorchain/thornode/x/thorchain/module.go:254 +0xf9c
github.com/cosmos/cosmos-sdk/types/module.(*Manager).EndBlock(_, {{0x3c76d90, 0xc00005a040}, {0x3c8a1e0, 0xc0f1d5d880}, {{0xb, 0x0}, {0xc1d7194510, 0x14}, 0xb3cbbc, ...}, ...}, ...)
github.com/cosmos/cosmos-sdk@v0.45.1/types/module/module.go:495 +0x202
gitlab.com/thorchain/thornode/app.(*THORChainApp).EndBlocker(...)
gitlab.com/thorchain/thornode/app/app_default.go:18
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc001701040, {0x40?})
github.com/cosmos/cosmos-sdk@v0.45.1/baseapp/abci.go:211 +0x1fe
github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc05d0a8420, {0xc05d0a8420?})
github.com/tendermint/tendermint@v0.34.14/abci/client/local_client.go:288 +0xdf
github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc0dbacb900?, {0x20?})
github.com/tendermint/tendermint@v0.34.14/proxy/app_conn.go:89 +0x24
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x3c789c8?, 0xc031df0ae0}, {0x3c805d0, 0xc05d0a3b50}, 0xc1c24f2b40, {0x3c876f8, 0xc05d0a2dc0}, 0xb3cbbb?)
github.com/tendermint/tendermint@v0.34.14/state/execution.go:327 +0x726
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc05c8f5d28, 0x7}}, {0xc01f4ba528, 0x14}, 0x490980, 0xb3cbbb, {{0xc0c4313aa0, ...}, ...}, ...}, ...)
github.com/tendermint/tendermint@v0.34.14/state/execution.go:140 +0x171
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc01bdfe700, 0x0)
github.com/tendermint/tendermint@v0.34.14/blockchain/v0/reactor.go:398 +0xb5a
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
github.com/tendermint/tendermint@v0.34.14/blockchain/v0/reactor.go:110 +0x7a
11:59AM INF github.com/cosmos/cosmos-sdk@v0.45.1/server/start.go:126 > starting ABCI with Tendermint
By contrast, v1.121.0 logs from prepared earlier (noting that the historical version was v1.116.0):
(v1.121.0 logs collapsible section)
12:27AM INF go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/state/execution.go:235 > committed state app_hash=5A96A22E8A420DAA6ED8BB43CEBB87398EA964E86E79E49B1606FC1337189D36 height=11783099 module=state num_txs=0
12:27AM INF app/x/thorchain/manager_network_v115.go:697 > maximum rune deployed from POL
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=DOGE
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=AVAX
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BNB
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=AVAX
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BNB
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=DOGE
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=GAIA
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=AVAX
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BNB
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=DOGE
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=GAIA
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=LTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=AVAX
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BNB
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=DOGE
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=GAIA
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=LTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=AVAX
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BNB
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=DOGE
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=GAIA
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=LTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=AVAX
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BNB
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=DOGE
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=GAIA
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=LTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=AVAX
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BNB
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=DOGE
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=GAIA
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=LTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=AVAX
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BNB
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=DOGE
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=GAIA
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=LTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BNB
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=DOGE
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=GAIA
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=LTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=AVAX
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BNB
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=DOGE
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=GAIA
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=LTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=AVAX
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=DOGE
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=GAIA
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=LTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=AVAX
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BCH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=BTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=DOGE
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=ETH
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=GAIA
12:27AM INF app/x/thorchain/manager_validator_v116.go:1069 > there is not fund for chain, no need for yggdrasil return chain=LTC
12:27AM INF app/x/thorchain/manager_validator_v116.go:1128 > some yggdrasil vaults (%d) still have funds
12:27AM INF go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/state/execution.go:235 > committed state app_hash=F294C47DAD06AB4464283EA67D51CD02429F68C15E10F9B1DB2624134B9B512B height=11783100 module=state num_txs=0
Further investigation is pending.
Update: Each .0 version backwards through v1.124.0 similarly panicked, but v1.123.0 did not.
At present, from the (*ValidatorMgrV116).RequestYggReturn
panic location and
https://gitlab.com/thorchain/thornode/-/commits/v1.124.0
I most suspect commit '[cleanup] remove yggs part 2/2 (thornode)'
, but it will take a while to sync from my most recent backup to a
thornode start --halt-height [uint]
height from which to test adjacent devlop-branch commits
and experiment with fixing this (perhaps including temporary log lines to more clearly see what's happening in that block).
addToBlockOut
v1.130.0 line 680 indicated by the panic message:
https://gitlab.com/thorchain/thornode/-/blob/v1.130.0/x/thorchain/manager_txout_v116.go#L680
telemetry.NewLabel("memo_type", memo.GetType().String()),
If I remember correctly, .GetType()
panics if a Memo interface is an empty interface rather than an interface-wrapped type with a MemoBase field.
That TxOutItem Memo:
https://gitlab.com/thorchain/thornode/-/blob/v1.124.0/x/thorchain/manager_validator_v116.go#L1083
Memo: NewYggdrasilReturn(ctx.BlockHeight()).String(),
https://gitlab.com/thorchain/thornode/-/blob/v1.124.0/x/thorchain/memo/memo_yggdrasil_archive.go#L44-49
func NewYggdrasilReturn(blockHeight int64) YggdrasilReturnMemo {
return YggdrasilReturnMemo{
MemoBase: MemoBase{TxType: TxYggdrasilReturn},
BlockHeight: blockHeight,
}
}
https://gitlab.com/thorchain/thornode/-/blob/v1.124.0/x/thorchain/memo/memo.go#L34
TxYggdrasilReturn // TODO remove on hard fork
It's currently unclear to me why there would be something strange about the Memo from v1.124.0 .
The line numbers were the same for v1.124.0:
(v1.124.0 logs collapsible section)
2:19PM INF app/x/thorchain/manager_network_v115.go:697 > maximum rune deployed from POL
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x78 pc=0x1ec009d]
goroutine 1 [running]:
gitlab.com/thorchain/thornode/x/thorchain.(*TxOutStorageV116).addToBlockOut(_, {{0x2f66d90, 0xc0001ae000}, {0x2f79a60, 0xc024877e00}, {{0xb, 0x0}, {0xc1378de030, 0x14}, 0xb3cbbc, ...}, ...}, ...)
/app/x/thorchain/manager_txout_v116.go:680 +0x3fd
gitlab.com/thorchain/thornode/x/thorchain.(*TxOutStorageV116).UnSafeAddTxOutItem(_, {{0x2f66d90, 0xc0001ae000}, {0x2f79a60, 0xc024877e00}, {{0xb, 0x0}, {0xc1378de030, 0x14}, 0xb3cbbc, ...}, ...}, ...)
/app/x/thorchain/manager_txout_v116.go:237 +0x1d6
gitlab.com/thorchain/thornode/x/thorchain.(*ValidatorMgrV116).RequestYggReturn(_, {{0x2f66d90, 0xc0001ae000}, {0x2f79a60, 0xc024877e00}, {{0xb, 0x0}, {0xc1378de030, 0x14}, 0xb3cbbc, ...}, ...}, ...)
/app/x/thorchain/manager_validator_v116.go:1091 +0xe29
gitlab.com/thorchain/thornode/x/thorchain.(*ValidatorMgrV116).recallYggFunds(_, {{0x2f66d90, 0xc0001ae000}, {0x2f79a60, 0xc024877e00}, {{0xb, 0x0}, {0xc1378de030, 0x14}, 0xb3cbbc, ...}, ...}, ...)
/app/x/thorchain/manager_validator_v116.go:1124 +0x7f0
gitlab.com/thorchain/thornode/x/thorchain.(*ValidatorMgrV116).EndBlock(_, {{0x2f66d90, 0xc0001ae000}, {0x2f79a60, 0xc024877e00}, {{0xb, 0x0}, {0xc1378de030, 0x14}, 0xb3cbbc, ...}, ...}, ...)
/app/x/thorchain/manager_validator_v116.go:272 +0x390
https://gitlab.com/thorchain/thornode/-/blob/v1.124.0/x/thorchain/manager_txout_v116.go#L676
That 'memo' for the .GetType()
:
memo, _ := ParseMemo(mgr.GetVersion(), item.Memo) // ignore err
https://gitlab.com/thorchain/thornode/-/blob/v1.124.0/x/thorchain/memo/memo.go#L227-244
func ParseMemo(version semver.Version, memo string) (mem Memo, err error) {
defer func() {
if r := recover(); r != nil {
// TODO: Remove conditional on hard fork, keeping the EmptyMemo setting.
if version.LT(semver.MustParse("1.116.0")) || version.GTE(semver.MustParse("1.122.0")) {
mem = EmptyMemo
}
err = fmt.Errorf("panicked parsing memo(%s), err: %s", memo, r)
}
}()
parser, err := newParser(cosmos.Context{}, nil, version, memo)
if err != nil {
return EmptyMemo, err
}
return parser.parse()
}
Because of the 'ignore err', and that it was a nil
interface rather than an EmptyMemo,
ParseMemo
returned a nil
interface and then the telemetry panicked when trying to do GetType()
to it.
Notably,
7bc0888d
with that before-116 after-121 was
'[Backwards-compatibility; V122-specific] Resolve block 11320435 sync failure',
implemented in V122 for a V113 sync failure,
and with no V116-through-V121 prompts prior to release V124.
A question then is what changed in V124 for its parsing to apparently fail while it didn't in V123.
https://gitlab.com/thorchain/thornode/-/blame/v1.123.0/x/thorchain/memo/memo_parser.go#L91-92
case TxYggdrasilReturn:
return p.ParseYggdrasilReturnMemo()
https://gitlab.com/thorchain/thornode/-/blame/v1.124.0/x/thorchain/memo/memo_parser.go#L116-120
case TxYggdrasilReturn: // TODO remove on hard fork
if p.keeper.GetVersion().GTE(semver.MustParse("1.124.0")) {
return EmptyMemo, fmt.Errorf("TxType not supported: %s", p.getType().String())
}
return p.ParseYggdrasilReturnMemo()
All seems to be in order at the moment, thus I will wait for the the re-sync from backup to complete before scrutinising more closely.
2024-04-18 update:
I have confirmed that ef1d46a1 (ygg removal 2/2) panics and its parent commit c5270695 does not.
What remains then is the 'why'.
ParseMemoWithTHORNames
:
https://gitlab.com/thorchain/thornode/-/blob/v1.131.0/x/thorchain/memo/memo.go#L264
parser, err := newParser(ctx, keeper, keeper.GetVersion(), memo)
ParseMemo
:
https://gitlab.com/thorchain/thornode/-/blob/v1.131.0/x/thorchain/memo/memo.go#L244
parser, err := newParser(cosmos.Context{}, nil, version, memo)
In other words, when any ParseMemo
call reaches something like
https://gitlab.com/thorchain/thornode/-/blob/v1.131.0/x/thorchain/memo/memo_parser.go#L120-124
case TxYggdrasilReturn: // TODO remove on hard fork
if p.keeper.GetVersion().GTE(semver.MustParse("1.124.0")) {
return EmptyMemo, fmt.Errorf("TxType not supported: %s", p.getType().String())
}
return p.ParseYggdrasilReturnMemo()
the p.keeper.GetVersion()
panics before it reaches the return EmptyMemo
.
To resolve the sync failure, this perhaps needs to reach the ParseYggdrasilReturnMemo
call in v1.116.0 .
My current thoughts then are to change the p.keeper.GetVersion()
calls to p.version
calls.
My impression is that this should not change historical ParseMemoWithTHORNames
behaviour,
since its p.keeper.GetVersion()
and p.version
are expected to be the same.
https://gitlab.com/thorchain/thornode/-/blob/v1.131.0/x/thorchain/memo/memo_parser.go#L40-45
return parser{
memo: memo,
txType: memoType,
ctx: ctx,
keeper: keeper,
version: version,
Other than those, the only p.keeper.
checks for nil
and so does not need to be changed:
https://gitlab.com/thorchain/thornode/-/blob/v1.131.0/x/thorchain/memo/memo_parser.go#L338-341
if p.keeper == nil {
return def
}
if p.keeper.THORNameExists(p.ctx, name) {
f45ae430 (a commit added to v1.130.0) appears to have resolved this block's sync failure!
If there are other block sync failures, they can hopefully be dealt with separately;
changing p.keeper.GetVersion()
to p.version
here is something I believe should definitely be done.
Specifically: !3523 (merged)
'[Backwards-compatibility] p.keeper.GetVersion()
-> p.version
to not panic when from ParseMemo
(which has a nil
keeper)'.