Failed outbounds not properly removed from ethereum (and evm?) signer cache
Example Inbound: https://thornode-v1.ninerealms.com/thorchain/tx/details/3CB3935B2C75707D954884ED251E10B6F568AF2CAA4283E6695E8E2F15D25D75 Outbound: https://etherscan.io/tx/0x563e3585278695650fde16ac31adc262b1b63bebd1c39c08a3dcee0dcd28d1e4
The upstream issue was #1796 (closed) and this particular case should be avoided with that resolution, but we should still ensure the failed outbound is removed from signer cache. It would seem https://gitlab.com/thorchain/thornode/-/blob/v1.126.0/bifrost/pkg/chainclients/ethereum/ethereum_block_scanner.go?ref_type=tags#L854 should do this, however it does not and there is no corresponding information in the logs.
Relevant logs from signer filtering for this outbound:
01/13 03:17:48 EST INF gitlab.com/thorchain/thornode/bifrost/pkg/chainclients/ethereum/ethereum.go:734 > broadcast tx with memo: OUT:3CB3935B2C75707D954884ED251E10B6F568AF2CAA4283E6695E8E2F15D25D75 to ETH chain , hash: 0x563e3585278695650fde16ac31adc262b1b63bebd1c39c08a3dcee0dcd28d1e4 module=ethereum service=bifrost
01/13 03:17:48 EST INF gitlab.com/thorchain/thornode/bifrost/signer/sign.go:653 > broadcasted tx to chain memo=OUT:3CB3935B2C75707D954884ED251E10B6F568AF2CAA4283E6695E8E2F15D25D75 module=signer service=bifrost txid=0x563e3585278695650fde16ac31adc262b1b63bebd1c39c08a3dcee0dcd28d1e4
01/13 03:39:11 EST INF gitlab.com/thorchain/thornode/bifrost/signer/sign.go:325 > Received a TxOut Array of {14252747 [{ETH 0x9c2E658ffC8ea7Fad00A4829Bd4B554e8a716f73 thorpub1addwnpepqtxdwxxysdr8hre4lkzqz9dd06g8t5s83cme67kdp6a839cyf2fr7y9445a 1572065270800 ETH.USDC-0XA0B86991C6218B36C1D19D4A2E9EB0CE3606EB48 OUT:3CB3935B2C75707D954884ED251E10B6F568AF2CAA4283E6695E8E2F15D25D75 [240000 ETH.ETH] 3 3CB3935B2C75707D954884ED251E10B6F568AF2CAA4283E6695E8E2F15D25D75 <nil> 0}]} from the Thorchain module=signer service=bifrost
01/13 03:39:12 EST INF gitlab.com/thorchain/thornode/bifrost/signer/sign.go:263 > Signing transaction height=14252747 module=signer num=0 service=bifrost status=1 tx={"aggregator":"","chain":"ETH","coins":[{"amount":"1572065270800","asset":"ETH.USDC-0XA0B86991C6218B36C1D19D4A2E9EB0CE3606EB48"}],"gas_rate":3,"height":14252747,"in_hash":"3CB3935B2C75707D954884ED251E10B6F568AF2CAA4283E6695E8E2F15D25D75","max_gas":[{"amount":"240000","asset":"ETH.ETH","decimals":8}],"memo":"OUT:3CB3935B2C75707D954884ED251E10B6F568AF2CAA4283E6695E8E2F15D25D75","out_hash":"","to":"0x9c2E658ffC8ea7Fad00A4829Bd4B554e8a716f73","vault_pubkey":"thorpub1addwnpepqtxdwxxysdr8hre4lkzqz9dd06g8t5s83cme67kdp6a839cyf2fr7y9445a"}
01/13 03:39:12 EST INF gitlab.com/thorchain/thornode/bifrost/pkg/chainclients/ethereum/ethereum.go:368 > transaction({Chain:ETH ToAddress:0x9c2E658ffC8ea7Fad00A4829Bd4B554e8a716f73 VaultPubKey:thorpub1addwnpepqtxdwxxysdr8hre4lkzqz9dd06g8t5s83cme67kdp6a839cyf2fr7y9445a Coins:1572065270800 ETH.USDC-0XA0B86991C6218B36C1D19D4A2E9EB0CE3606EB48 Memo:OUT:3CB3935B2C75707D954884ED251E10B6F568AF2CAA4283E6695E8E2F15D25D75 MaxGas:[240000 ETH.ETH] GasRate:3 InHash:3CB3935B2C75707D954884ED251E10B6F568AF2CAA4283E6695E8E2F15D25D75 OutHash: Aggregator: AggregatorTargetAsset: AggregatorTargetLimit:<nil> Checkpoint:[] Height:14252747}), signed before , ignore module=ethereum service=bifrost