feat(thorchain): keep gas in vaults for the tokens they hold (VaultTokenGasReserve)
🔎 Visual review of2fafbf29: explainer, review flags and prompts to ask an AI about the change
Why
A vault can run out of a chain's gas asset while still holding that chain's tokens. A large gas-asset outbound (TRX, ETH, ...) holds back only its own MaxGas, so it can drain a vault that holds tokens. That vault is then skipped for every token outbound, and nothing moves gas back to it, so its tokens are stuck until someone donates gas on L1. On mainnet today, two TRON vaults hold 52k and 377k USDT with 18 and ~600 TRX.
Refs #3445. This MR covers gaps 1 and 2 from the issue; gaps 3 and 4 are follow-ups (see below).
What changes
All behavior changes are behind a new Mimir key, VaultTokenGasReserve, which defaults to 0 (off). With the key off, vault selection behaves exactly as before.
When VaultTokenGasReserve is above 0:
- Gas reserve for tokens. When vaults are chosen for a gas-asset outbound, each vault keeps back
VaultTokenGasReserve × MaxGasof the gas asset per non-gas token it holds on that chain. - All pending outbounds on the chain count. A vault's available balance now subtracts every queued outbound on the chain, not only those of the same asset. This applies to gas-asset and token outbounds. A token outbound's gas check therefore sees gas already committed to queued gas-asset outbounds from the same vault, and vice versa. On mainnet, vault 58h5 was recorded at 36,350 TRX while 35,777 TRX of outbounds were queued from it.
- Fallback to the Mimir-off selection. Selection with points 1 and 2 applied is accepted only if it fills the whole outbound without a sub-fee fragment. For a gas-asset outbound, a sub-fee fragment is an output no larger than the outbound fee plus dust. For a token outbound, it is an output no larger than the fee.
prepareTxOutItemwould drop such a fragment after the inbound was already debited. If the selection fails either test, it runs exactly as with the Mimir off: no reserve, and vault balances net only of pending outbounds of the same asset. The chain-wide deduction is dropped as well, because gas committed to other assets' outbounds can exclude a vault the Mimir-off selection would use and split off a sub-fee fragment. So every outbound is either paid with the key's rules and no sub-fee fragment, or scheduled exactly as with the key off. The key never blocks an outbound, splits it into a remainder, or leaves it with a dropped tail.
Ragnarok picks vaults as if the key were off. It holds no reserve and subtracts only same-asset pending outbounds. Ragnarok zeroes any residual the vaults can't pay instead of sending it to recovery. If a vault were skipped because its gas is committed to other assets' outbounds, the LP would lose that part of the payout after withdraw() had already debited the pool.
Code:
x/thorchain/manager_txout_current.go: the vault loop inDiscoverOutboundsmoves, unchanged, intoallocateOutbounds, so it can run a second time without the key's rules. Use--color-movedto review it. The new logic is:- the reserve block;
nonGasTokenCount;hasUnpayableFragment/isUnpayableFragment, the sub-fee fragment check (fee plus dust threshold for gas-asset outbounds, fee alone for tokens);- the vaults-necessary sort moves, unchanged, into
sortOutboundVaults, so the first pass and the fallback each sort their own vault view; discoverOutboundstakes two vault views. The first pass uses vaults net of every pending outbound on the chain, with the reserve applied for gas-asset outbounds. If that pass can't fill the outbound without a sub-fee fragment, the rest of the function (unchanged from develop) runs onfallbackVaults. Those are net only of same-asset pending outbounds, which is exactly the Mimir-off selection;- in
prepareTxOutItem: the Mimir read, the Ragnarok memo check that sets the key to 0, the choice of pending list, and, with the key on, the fallback view (vaultsNetOfPendingdeducts the same-asset subset of the chain-wide list from copies of the vaults).
x/thorchain/keeper: newGetPendingOutboundsByChain. It shares a body withGetPendingOutbounds. Only the item filter differs, so it walks the same txout store keys.constants: newVaultTokenGasReserve(default 0) and its stringer entry.docs/mimir.mddocuments the key. It applies only to outbounds whose vault is chosen by selection. Refunds kept on the observed vault, churn migrations, rescheduled outbounds and Ragnarok outbounds keep their existing selection, with no reserve and no chain-wide pending deduction.
Consensus notes
- With the key off, every outbound that has to choose a vault now reads
VaultTokenGasReserveonce. That is one more KV read, sogas_usedchanges for transactions that schedule an L1 outbound. The outbound memo is also parsed to detect Ragnarok, which costs CPU but no store reads. Vault choice and all state stay the same. - With the key on, the chain-wide pending list costs no extra reads, because it walks the same keys as the same-asset list. The fallback's same-asset list is filtered from it in memory, so it costs no reads either.
- Determinism: slices only, no map iteration, no floats. The reserve uses
cosmos.Uintmath.
Choosing a value
MaxGas on TRON is currently about 25.4 TRX. For example, VaultTokenGasReserve=20 keeps about 508 TRX per token in each vault, enough for about 35 USDT transfers at the current ~14 TRX each. The value is per token per vault, and it only applies when a vault is picked for a gas-asset outbound.
Tests
- Unit tests (
TxOutStoreSuite) cover:- the reserve on, splitting an outbound across two vaults;
- the key off (as before);
- the fallback when the reserve can't fit, with no security event;
- the fallback when the reserve split would leave a sub-fee tail, and a tail of exactly fee + dust keeping the split;
- token outbounds, which hold nothing back;
- Ragnarok in
DiscoverOutbounds, which sweeps without a reserve; nonGasTokenCount;prepareTxOutItemsending a token outbound to another vault when its vault's gas is committed to a queued ETH outbound (key on), and keeping it on that vault with the key off;prepareTxOutItemwith a Ragnarok memo, key on and off: the vault with gas committed elsewhere still pays its 1000 USDT share, and the other vault pays the remaining 200;- the fallback matching the Mimir-off selection exactly:
TestDiscoverOutboundsTokenGasReserveFallbackMatchesMimirOff(one sub-fee fragment) andTestDiscoverOutboundsReserveFallbackMultiFragmentMatchesMimirOff(vaults of 100, 0.0287 and 0.0287 ETH); - end to end in
prepareTxOutItem:TestPrepareTxOutItemReserveFallbackKeepsFragment(sent + fee equals the amount requested, key on and off) andTestPrepareTxOutItemReserveFallbackMatchesMimirOff. The latter uses a queued USDT outbound that commits the token vault's gas, two gas-only vaults, and the pool with and without LP units. It checks the review's case (a fragment between the fee and fee + dust that the earlier fold dropped) and a split into several fragments. Key-on outputs equal key-off outputs, each fits its vault with MaxGas, and the one-vault case pays the request less one fee (no fee with zero units). All threeMatchesMimirOfftests fail against235961de7e; - token outbounds through
TryAddTxOutItem:TestTryAddTxOutItemTokenFragmentNotDropped. Gas committed to a queued ETH outbound excludes the one vault that could pay a USDT outbound whole, so the chain-wide pass would split off a USDT fragment smaller than the fee. Selection falls back to the Mimir-off choice, and every unit of the request is either sent or taken as the fee, with nothing left to recover.
- Keeper tests:
GetPendingOutboundsByChainvsGetPendingOutbounds. - Coverage of changed lines (measured before
0cad696f30): 89/94 inx/thorchain/manager_txout_current.go, 14/14 inx/thorchain/keeper/v1. The 5 uncovered lines are in the moved vault loop. - New regression suites in
test/regression/suites/outbound/, using a two-vault ETH state (one vault holds USDC):token-gas-reserve-off: a 12 ETH withdraw is paid by the token vault alone.token-gas-reserve: the token vault pays 10.184 ETH and keeps ≥ 500 × MaxGas; the gas-only vault pays the rest. Both outbounds are observed and the final vault balances asserted.token-gas-reserve-fallback: the reserve can't fit, so the token vault pays everything and no security event is emitted.token-gas-reserve-tail: a 10.196 ETH withdraw is paid by the token vault alone. With the reserve, it would split off a 0.0056 ETH tail, below the 0.0064 ETH fee.token-gas-reserve-pending-fragment: a queued USDC outbound commits the token vault's gas, and a 14.986 ETH withdraw falls back without the reserve. It selects as with the key off, so the 0.0052 ETH fragment a chain-wide view would leave on the gas-only vault never arises. The token vault pays it all (1497960000 after the fee) with no security event.
- Local runs:
make test-regression RUN=token-gas-reserve: 5/5 pass (token-gas-reserve-pending-fragmentexported withEXPORT=1; the other four goldens are unchanged). Rerun at1777c43d1e(fallback = Mimir-off selection): 5/5 pass with all five goldens unchanged.go test -tags=mocknet ./x/thorchain/... ./constants/...: pass.make lint-ci: no issues.make generate: clean.
- The full regression suite was not run locally; CI runs it. The new fixtures are only those added with the new suites.
Not in this MR (follow-ups)
inbound_addressesand inbound routing ignore gas need, so inbound gas doesn't go preferentially to gas-starved vaults.- Churn migrations name their source vault and skip the gas check. A retiring vault already below MaxGas still can't migrate its tokens.
- An active vault that is already out of gas still needs an L1 donation (
DONATE:<CHAIN>.<GAS>). This MR prevents vaults from being drained in the first place; it does not refill one that already ran out.