fix(thorchain): keep a solvency halt until the vault's gap is covered

🔎 Visual review of c0e22247: explainer, review flags and prompts to ask an AI about the change

Problem

When a vault is found short on a chain, processSolvencyAttestation flags it and halts the chain. The halt stops Bifrost scanning the chain, and the signer refuses to sign while the scanner is unhealthy, so nothing scheduled during the halt is sent.

insolvencyCheck still nets every pending outbound out of the vault's balance, on the assumption that it may already have left the wallet. Outbounds keep being scheduled during the halt (swaps into the chain still complete). Once they add up to more than the gap, the vault looks solvent: its flag is cleared and the chain unhalts while the vault is still short. Signing then drains the queue, the gap reappears and the chain halts again. A vault that is really short cycles like this for as long as the queue refills, and each unhalted window pays out.

We saw this on mainnet ETH on 2026-10-31: the chain unhalted three times in about 80 minutes with the vault still short, each time with the queue covering the gap.

Change

  1. While the chain is solvency-halted, insolvencyCheck nets out only the outbounds that were pending when signing stopped. That point is the cutoff: SolvencyHalt<Chain>Chain plus SolvencyHaltSigningLagBlocks, a new constant (default 3) that Mimir can override. The lag covers Bifrost's delay in seeing the halt, since its scanner re-reads Mimir about once per THORChain block. The pending set is read from blocks before the cutoff:

    • an item still unsent counts while its signing period at the cutoff lasts;
    • an item LackSigning replaced at or after the cutoff (OutHash BlankTxID) counts while its copy is still unsent. LackSigning adds the copy at the item's height plus its signing period, rounded up to RescheduleCoalesceBlocks, so the copy is looked up there, matched on chain, InHash, destination and coin (the vault can differ, since LackSigning reassigns), and followed if it was replaced again. An observed copy has already debited the vault, and a missing one (treasury recovery, a SOL reissue at another height) means nothing is in flight, so in both cases the original doesn't count. Migrations and consolidations are re-planned rather than copied, so a replaced one doesn't count either.

    Blocks from the cutoff on hold only outbounds released since and copies of earlier ones, none of which Bifrost could have sent, so they don't count.

  2. The halt keeps the height it began at. It is no longer refreshed to the current block while vaults stay insolvent, because the cutoff is dated from it. This also dates a scheduled halt correctly once it takes effect.

  3. A solvent report no longer lifts the halt while another Active or Retiring vault holding funds on the chain is flagged insolvent. Only that vault's own solvent report clears its flag. Silence doesn't show recovery, because a vault that is still short goes silent too when its reports stop reaching consensus. The per-vault flag now records the vault's latest insolvent report rather than its first.

    • SolvencyHaltHoldBlocks, a new constant, defaults to 0 (without limit). Mimir can set a window as a way out if a flagged vault can't report at all.
    • A report that clears its vault's flag doesn't also lift the halt; the next solvent report does. Consensus on a recovered vault's report and another vault's first insolvent report can land in either order, and the insolvent one has to run under the halt, or it nets out the backlog the halt built.
    • Vaults with no booked funds on the chain and Inactive vaults are left out, since Bifrost doesn't report them.
    • This does, across blocks, the job the refresh did only within one block.
  4. Bifrost reports a fixed vault individually. While the chain is halted and every vault could be read, each chain client also reports a solvent vault that THORChain still has flagged, alongside the insolvent ones (runners.RecoveredVaultReports). Before, a solvent vault was reported only in an all-clear, which another vault's unobserved outbound can block, so a fixed vault would have held the halt with no way to clear its flag. Solvent vaults that aren't flagged stay quiet.

  5. docs/mimir.md: when SolvencyHalt<Chain> lifts, what its value means, and the two new constants.

What doesn't change

  • When the chain isn't solvency-halted, insolvencyCheck nets out every pending outbound exactly as before. That includes the report that starts the halt.
  • Other halt mechanics: a manual 1 halt has no height, so it nets out everything as before. Future-height halts and StopSolvencyCheck* work as before.

Risk: consensus path

  • THORNode and Bifrost have to ship together. With new THORNode and old Bifrost, a fixed vault stays silent while another vault's unobserved outbound blocks the all-clear, and its flag holds the halt.
  • It needs a coordinated release. Solvency processing now reads the halt key, and while halted it also reads the constants, a longer txout window and the copies' blocks. An insolvent report now writes its vault's flag every time, not just on the first. The halt/unhalt decisions themselves also change.
  • A recovered chain lifts one solvency cycle later than before, since the report that clears the last flag doesn't lift the halt. That's about one block on BTC.
  • A real shortfall now keeps the chain halted until it is resolved: by errata, a write-down or a top-up of the vault, or by node vote (next bullet). Before, the halt lifted once the queue grew past the gap.
  • Lifting a halt by vote: StopSolvencyCheck<Chain> alone doesn't lift it. processSolvencyAttestation returns before any flag or unhalt logic, and IsChainHalted and Bifrost's pause check read SolvencyHalt<Chain>Chain without consulting it, so the stop freezes both the halt and the flags. Develop returns early the same way; there, the backlog lifted the halt on its own and hid it. To lift a halt by vote (all three keys are operational, 3 votes each):
    1. Set StopSolvencyCheck<Chain> to 1, so the next insolvent report doesn't halt the chain again.
    2. Set SolvencyHalt<Chain>Chain to 0.
    3. Set each flagged vault's SolvencyHaltVault-<pubkey>-<chain> to 0 if it should take deposits again. Its flag is frozen too, and keeps it out of inbound election. Then remove the stop once the shortfall is resolved. If a vault is still short, its next report halts the chain again.
  • One unreadable vault delays a fixed vault's report, since it's sent only once every vault could be read.
  • A flagged vault that can't report at all holds the halt until it can. Setting SolvencyHaltHoldBlocks by Mimir is the way out, with care: once the window passes, a vault that is still short no longer holds the halt, which is develop's behaviour today.
  • SolvencyHalt<Chain>Chain now shows when the halt began, not the last insolvent report.
  • Upgrade: a halt already running at the upgrade carries the height the old code last refreshed it to. Its cutoff is later than the real start, so outbounds queued earlier in that halt still net out. Halts that start after the upgrade are dated correctly.
  • What still isn't covered: two masking sources remain, both fixed at the moment of the halt rather than growing with it. One is outbounds released before the cutoff that weren't actually sent. The other is deposits reaching the wallet unobserved while the scanner is paused.
  • A race that remains: a vault that was never flagged and nets solvent (say, an unobserved outbound) can still lift the halt just before another vault's first insolvent report lands. That report then nets out the backlog. This is develop's behaviour too. Only a flag-clearing report is held back here.
  • Edges that fail closed: a SOL reissue's original isn't followed, since its copy lands at another height. A migration in flight at the halt and pruned as stale during it stops counting. A copy is looked up with the current SigningTransactionPeriod and RescheduleCoalesceBlocks, so a Mimir change to either after LackSigning ran misses it. In each case the vault looks short and the halt holds.

Relation to other MRs

  • !5304: its gate already ignores pending outbounds when deciding to release a parked inbound. This MR stops the chain halt itself from lifting the same way. The two are complementary.
  • !4469 (merged): two of its steps no longer hold. A solvent vault reporting first in a new block now leaves the halt in place while the other vault is flagged, and the halt height isn't refreshed.

Testing

  • New tests in handler_solvency_helpers_test.go:
    • TestInsolvencyCheck_SolvencyHaltNetsOutOnlyEarlierOutbounds (9 cases): not halted, before, within and after the lag, the lag set by Mimir, a cutoff not yet reached, a manual halt, and a future halt;
    • TestInsolvencyCheck_SolvencyHaltCountsReplacedOutbounds (18 cases, with LackSigning's real copy heights):
      • copies unsent, landed before the cutoff (no double count), coalesced past it, observed after it (no double deduction), missing (treasury recovery), and at another height (a SOL reissue);
      • a replaced migration;
      • reassignment, for both the old and the new vault, and another vault's original;
      • a later sibling, both ways;
      • replaced twice (unsent and observed);
      • two identical originals with two copies and with one;
      • expiry dated from the cutoff, both ways;
    • TestInsolvencyCheck_SolvencyHaltCoalescedCopyOnShortWindowChain: BTC;
    • TestInsolvencyCheck_LackSigningRescheduleStillNetsOut: runs the real LackSigning, with and without coalescing;
    • TestProcessSolvencyAttestation_HaltBacklogDoesNotLiftHalt: the mainnet sequence;
    • TestProcessSolvencyAttestation_ScheduledHaltDatesTheCutoff: crosses the activation height;
    • TestProcessSolvencyAttestation_FlaggedVaultHoldsHaltUntilItReportsSolvent: Huginn's all-clear sequence: a silent vault holding without limit, a vault still reported, a fixed vault's solvent report before and after the other vault's, and a window set by Mimir;
    • TestProcessSolvencyAttestation_RecoveredReportKeepsHaltForNewlyShortVault: a recovered report and a newly short vault's first report, in both orders, in one block and two, with and without a backlog;
    • TestProcessSolvencyAttestation_SilentShortVaultKeepsBacklogFromCoveringItsGap: a short vault silent past any window, then reporting with a backlog;
    • TestProcessSolvencyAttestation_FlaggedVaultKeepsHalt (Active and Retiring), …_StaleFlagsDoNotKeepHalt (a written-down vault and an Inactive vault), and TestFlagVaultInsolvent_TracksLatestInsolventReport.
  • TestProcessSolvencyAttestation_InsolventButAlreadyHalted now expects the halt height kept rather than refreshed.
  • New in Bifrost: TestRecoveredVaultReports (runners) and TestReportSolvencyWhileHaltedReportsRecoveredVaultAlongsideInsolvent (Monero).
  • Mutation checks (at 224ca0cf): 25 mutations, each breaking one condition, and every one fails a test. The cases include the observed copy, the missing copy, coalescing, the copy chain, copies claimed twice, the hold window and its 0 setting, and the flag refresh. The cutoff < pendingAt guard in insolvencyCheck only keeps the logged height sane, so it was left out.
  • At d0602e4c: go test -tags=mocknet passes for ./x/thorchain/..., ./constants/... and the runners package, against a compile-only stub. Three mutations of the new check each fail tests. Four existing tests that asserted the clearing report also lifts now expect the lift from the next report; one of them, TestProcessSolvencyAttestation_ClearsVaultInsolventFlag, is on develop. The chain clients aren't changed here, and neither regression suite that sends solvency reports has a flagged vault report solvent, so test-regression doesn't exercise this lift path. The pipeline passed with no golden changes. c0e22247 changes only a comment and a log line; the handler suite passes on it.
  • At d4fd918b: the pipeline passed, including unit-tests, unit-tests-race-bifrost, lint and test-regression, and restoring a 1200-block default fails both new handler tests.
  • At 224ca0cf: go test -tags=mocknet ./x/thorchain/... ./constants/...: all packages pass; the x/thorchain suite has 1746 passed and 2 skipped. Regression: all 193 suites pass, with no retries and every block and state export matching develop's goldens.
  • golangci-lint 1.63.4 (repo config, new code), gofumpt v0.5.0, go vet, and scripts/test-integrity-guard.py are all clean. The constants' stringer file was regenerated with the Makefile's stringer@v0.28.0.

tss-lib-private isn't public, so this was built against a compile-only stub. The stubbed symbols are linked, but they aren't on any path this change touches.

Edited by Night Watch

Merge request reports

Loading
Loading