Cancelled ETH/EVM transactions' TxOutItems remain stuck at too-low gas forever until either sufficient or swallowed
2024-04-29 update:
Update's update: I was initially alarmed by the context, but the item was gratifyingly soon successfully broadcast, rather than being stuck at too-low gas indefinitely.
I drop this to priority 2 for now and will continue to pay attention.
https://thornode-v1.ninerealms.com/thorchain/tx/status/644D862D02B997C7248F9B03EA82E07AC9985CE70B0D283BE725BAE95C384275
https://etherscan.io/gastracker
(2024-04-29 update initial contents)
https://discord.com/channels/838986635756044328/1187086349283627151/1234479024819011617
[15753684]
Rescheduled Outbound: 10722.310957 ETH.USDC ($10,722.31)Age: 300 blocks (30 minutes)
Inbound Memo:SWAP:eth.usdc-0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48:0x9c2E658ffC8ea7Fad00A4829Bd4B554e8a716f73:1071205966160
Outbound Memo:OUT:644D862D02B997C7248F9B03EA82E07AC9985CE70B0D283BE725BAE95C384275
Vault:g500
=>g500
Gas Rate:1
=>1
Max Gas:150000
=>150000
"chain": "ETH",
"address": "0xaf102d2b68e1ce1b11bfd7cd7c4f18717359bb0c"
https://etherscan.io/tx/0x6d5884e92f2739f35a5605e701eb3f3dc94f32d24614ef5773e61ca85d8780bb
Cancel transaction (to self, 0x input data, replaces nonce 1046 other transaction).
https://etherscan.io/tx/0x3dafc4cf477dd8537352a1324a617b599f896e4d4fd2a690005ce9e3585bca45
3 memo string OUT:644D862D02B997C7248F9B03EA82E07AC9985CE70B0D283BE725BAE95C384275
Same memo, gas price 10 Gwei (0.00000001 ETH)
.
"consensus_height": 15753384,
"finalised_height": 15753384,
"outbound_height": 15753384
https://thornode-v1.ninerealms.com/blocks/15753384
"time": "2024-04-29T11:48:06.361500349Z",
https://thornode-v1.ninerealms.com/thorchain/version?height=15753384
"current": "1.131.1",
This is to say, this was long after block 15556059 (2024-04-15)'s v1.130.0 which included !3509 (merged)
'[ethereum] Improve Dynamic Transaction Fee Handling, Dynamic Fee Outbounds, Allow Gas Limit Buffer'
and appears to be a new victim of this Issue, namely stuck at the same low (here minimum) gas rate until either that is sufficient or the TxOutItem is swallowed by the vaults.
THORNode's perceived ETH gas rate (from MsgNetworkFee broadcasts) was indeed 10 gwei in the inbound block.
https://thornode-v1.ninerealms.com/thorchain/inbound_addresses?height=15753384
"chain": "ETH",
"gas_rate": "10",
"gas_rate_units": "gwei",
2024-04-26 update: The example transaction
https://thornode-v1.ninerealms.com/thorchain/tx/status/B171867035950E57526222212C61E677C0D6E861A6DAA3DAF710B89213D48925
was swallowed through temporary setting of MaxOutboundAttempts
to 150 in block 15588472 (set back in block 15588740).
|
https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=15588471
5 queued items, of which it was the second.
https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=15588720
3 queued items, of which it was the first.
https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=15588722
Zero queued items.
Discord context:
https://discord.com/channels/838986635756044328/1220447176971386932/1228793666081001606
https://thornode-v1.ninerealms.com/thorchain/inbound_addresses?height=15531800
"chain": "ETH",
"gas_rate": "160",
"outbound_fee": "3300000",
"gas_rate": "160",
"gas_rate_units": "gwei",
0.033 Ether total, 0.0000016 Ether per gas unit,
in short 206,250 gas units allowed.
https://thornode-v1.ninerealms.com/thorchain/queue/outbound?height=15531800
Two ETH.ETH outbounds stuck at
"gas_rate": 1,
, both having been in the outbound queue for over nine hours at time of writing.
B171867035950E57526222212C61E677C0D6E861A6DAA3DAF710B89213D48925
39B73726958719D592BED657CE3DF1BDB229EE7D677DD92A0E1084D50DD17F1C
Each were also stuck at 0.0015 Ether allowed MaxGas.
By contrast, a recent ETH.ETH outbound:
https://thornode-v1.ninerealms.com/thorchain/tx/details/A00873D2BFB5BBB5E0B5EE99E694947B0DBB6D810A7A2B1585B77AEE288F78C7
0.01792620 Ether gas cost (transaction fee).
This specific gas_rate
1 should hopefully be fixed for future outbounds by V131's !3509 (merged)
'Improve Dynamic Transaction Fee Handling, Dynamic Fee Outbounds, Allow Gas Limit Buffer',
but I am concerned about the implications for any uncommonly-low gas period.
For one example transaction in particular:
https://thornode-v1.ninerealms.com/thorchain/tx/details/B171867035950E57526222212C61E677C0D6E861A6DAA3DAF710B89213D48925?height=15532000
One actions
, no out_txs
, planned gas_rate
1, "finalised_height": 15522754,
(15 hours earlier).
Instant-observations (17 signers) tx:
https://thornode-v1.ninerealms.com/thorchain/tx/details/6d087256b9e8cff5c908f07bcbb365d1e9a5e31a9f637b886b1f4e686f5e88ea
https://etherscan.io/tx/0x6d087256b9e8cff5c908f07bcbb365d1e9a5e31a9f637b886b1f4e686f5e88ea
From 0x..3167
, Nonce 6830, Gas Price 10 Gwei.
# | Name | Type | Data |
---|---|---|---|
0 | to | address | 0x9c2E658ffC8ea7Fad00A4829Bd4B554e8a716f73 |
1 | asset | address | 0x0000000000000000000000000000000000000000 |
2 | amount | uint256 | 5842633130000000000 |
3 | memo | string | OUT:B171867035950E57526222212C61E677C0D6E861A6DAA3DAF710B89213D48925 |
Cancel tx (unstuck.go
), prompted by that the transaction was not observed in a block (from too-low gas):
https://thornode-v1.ninerealms.com/thorchain/tx/details/65fadd4c32dc889d516e502dee1b8f056ee9e6b4df21aa33487da89452f3f96b
https://etherscan.io/tx/0x65fadd4c32dc889d516e502dee1b8f056ee9e6b4df21aa33487da89452f3f96b
From 0x..3167
, Nonce 6830, Gas Price ~35.7 Gwei.
As a gas asset (ETH.ETH) outbound Bifrost does not exceed THORNode-stipulated MaxGas, which is consistent with my personal preference.
https://gitlab.com/thorchain/thornode/-/blob/v1.130.0/bifrost/pkg/chainclients/ethereum/ethereum.go#L604-605
} else if !tx.Coins[0].Asset.IsGasAsset() {
scheduledMaxFee = scheduledMaxFee.Mul(scheduledMaxFee, big.NewInt(c.cfg.TokenMaxGasMultiplier))
This is the needsNewVault
code which stops THORNode from ever rescheduling the TxOutItem with a higher GasRate:
https://gitlab.com/thorchain/thornode/-/blob/v1.130.0/x/thorchain/manager_slasher_current.go#L750-756
// if a super majority of vault members have observed the outbound,
// then we should not reschedule. If a vault says it sent it, it
// sent it and shouldn't get another vault to send it (potentially
// a second time)
if count > 0 && HasSuperMajority(count, len(vault.Membership)) {
return false
}
In summary, Bifrosts cancel the transaction because its gas rate was too low to be included in a block,
but then THORNode keeps the low gas rate and MaxGas unchanged because of the pre-cancellation instant observations.
(THORNode currently cannot see that an observed cancel transaction had the same nonce as a GetObservedLink
-origin partially-observed ObservedTxOutVoter.)
On the one hand, if a TxOutItem is stuck with a too-low gas rate that cannot be updated, there is concern that it might remain that way until swallowed by MaxOutboundAttempts
.
On the other hand, are there double-spend concerns if an external chain has problems and a cancel tx is unable to cancel an earlier tx before a double-spend takes place?
I believe the Bifrost code does currently allow the same vault to re-try signing a TxOutItem after it has been cancelled from too-low gas rate,
but it seems undesirable to me for THORNode to hold to that too-low gas rate forever
until either the gas rate is sufficient for block inclusion
or the TxOutItem is swallowed by MaxOutboundAttempts
.
// remove the outbound from signer cache so it can be re-attempted
e.signerCacheManager.RemoveSigned(tx.Hash().Hex())
Depending on others'(/consensus) preferences it may be possible to close this Issue with no code change;
however, at the moment I feel the question worth raising and discussing.