EVM transactions stuck in outbound queue since gas estimate exceeds limit
$ tci network stuck-outbounds --threshold 3000
asset amount age scheduled vault vault_balance to_address in_hash
AVAX.USDC 16.822936 4582 14509008 4sra 210709.499541 ...A6Aa C1059C2ACA88BEDFCDF06CAC6F978C77C1D853FC5F74D4C42184EA75296B5D38
AVAX.USDC 780.379529 7794 14508796 m6s2 241744.316087 ...d2c2 C3992E8C1970545C0CDED9F3C381F1FD6521F37F2B79BAE647C02630F1EF217D
Currently Bifrost reports a static limit of 80k gas units in network fee observations. These units multiplied by the rate in the network fee are used to determine the "max gas" (actually a fee) allowed to an outbound. When Bifrost signs an outbound it uses the max of the current gas rate and the max set in the outbound, but if the estimated gas units are over the 80k in the network fee observation then the estimated fee is always over the "max gas" (fee) set in the outbound.
Bifrost used to be allowed lower the gas rate on the outbound transaction, which would allow sending these with a lowered gas rate. The change !3333 (merged) prevented Bifrost from lowering the gas rate dynamically to avoid outbounds becoming stuck in mempool due to broadcasting with a lower gas rate, but it was an assumed condition that all outbounds could be satisfied with under MaxGasLimit
gas units. We should update the reported gas limit to 100k, which should be sufficiently over all outbound gas units in recent observation.
Separately, there is a caveat in the gas manager GetGasRate
and GetMaxGas
functions, where due to integer math the inferred gas limit of the outbound can be higher than 80k. This loophole has caused some outbounds to succeed after !3333 (merged) even when they exceed the 80k gas limit. Probably no need to fix, but noting for relevance:
>>> limit = 80000
>>> gas_rate = 4
>>> # GetMaxGas
>>> gas_rate * limit * 3 // 2
480000
>>> # GetGasRate
>>> gas_rate * 3 // 2
6
>>> # inferred gas limit
>>> 480000 / 6
80000.0
>>> gas_rate = 3
>>> # GetMaxGas
>>> gas_rate * limit * 3 // 2
360000
>>> # GetGasRate
>>> gas_rate * 3 // 2
4
>>> # inferred gas limit
>>> 360000 / 4
90000.0