fix(tss): repair init vault chain set from late keygen reports
Problem
The churn at mainnet block 27820568 was aborted with a security event:
init vault chain set mismatch, skipping rotationTwo of five new Asgard vaults came out without SOL, so tryRotateInitVaults correctly refused to activate an asymmetric set — vault.Chains drives inbound addresses, migration targets and routers, so a chain-less Asgard would be permanently unusable for that chain.
Tx: B964D108F9DB7C53B596828DD1F826AD003D772FC256101364E2B259FCF6F535
Root cause
Replaying the on-chain vote order pins it exactly:
- Each node's
MsgTssPoolcarries its bifrost's enabled chain list (Signer.keygenChainsForType) — pure config (Disabled/OptToRetire), not liveness. ConsensusChains()keeps a chain only ifHasSuperMajority(count, len(PubKeys))— for 18 members, 12 votes.- During a churn that promotes Ready nodes,
asgardKeygenRequiresCompleteConsensusis false, so the vault finalizes at first supermajority = 12 of 18 rather than the usual 17 of 18. chains := voter.ConsensusChains()therefore runs against 12 reports, so a chain effectively needs unanimity among the first 12 reporters. In both failing vaults exactly one of the first 12 had SOL disabled → SOL sat at 11, one short, and was dropped. Report 13 would have carried it.if shouldFinalizeVault && !VaultExists(...)meant reports 13-18 — which all did include SOL — never revisited the chain set.
Both failing vaults, from the block data:
| vault | finalized | dissenter | later reports |
|---|---|---|---|
...qwvfttet |
report 12, block 27820468 | thor1sqf8fju… (Ready) |
13-18 same block; rotation not attempted until block 27820568 |
...q0xxj4l0 |
report 12, block 27820568 | thor1xjm3wnx… (Active) |
13-17 same block; rotation attempted inline in that message |
That table is why both halves of the fix are load-bearing: a chain refresh alone fixes the first vault, but not the second, where rotation had already run.
Change
1. Refresh an InitVault's chain set as late reports arrive. New refreshInitVaultChains, taken when an Asgard keygen-success report lands for a vault that already exists. ConsensusChains() is monotone — Sign only appends and the denominator len(PubKeys) is fixed — so a refresh can only add chains. The vault's own chains are merged in and the result re-sorted, so an AddFunds observation is never undone and initVaultsAgreeOnChains's positional compare stays valid. Routers are recomputed with the set (they are derived from it at creation, and are otherwise only backfilled when entirely nil).
2. Re-attempt rotation, gated on the chain set actually changing — the widening is what makes the init vaults agree. Idempotency comes from rotateReadyKeygenVaultFamilies only acting on InitVault vaults, plus an explicit status guard in the helper: RotateVault is not safe to call twice, a second call matches the now-Active vault against its own membership and would set it Retiring.
The path is deliberately surgical — no re-emitted keygen-success event, no keygen metric, no re-run check-signature quorum, and voter.BlockHeight is untouched (restamping would re-open the ObservationDelayFlexibility window and refund slash points to arbitrarily late reporters). judgeLateSigner already ran for late reports before this change and still does; no behaviour change there.
Rejected: changing ConsensusChains's denominator to len(Signers). A chain backed by 8 of 12 early reporters is only 8 of 18 overall and could never assemble a 12-of-18 keysign — that trades a churn stall for an unsignable chain.
3. Bifrost warning. At keygen time, diff the local chain list against the chains the active vaults serve and log an error per missing chain, so an operator sees the misconfiguration before it stalls a churn. Log-only; GetAsgards() errors are swallowed.
4. Cleanup. initVaultsForKeygenBlock had no production caller — stale init vaults are already swept to Inactive by TriggerKeygen. Removed with its test.
Testing
New TestLateKeygenReportWidensInitVaultChains replays the mainnet sequence: two keygen groups (one group alone would rotate immediately and never reach the new path), 8 members, one dissenter inside the early reporters. It asserts the vault is created without the chain, that the mismatch security event fires and nothing rotates, then that the next report widens the set, picks up the router, and rotates both vaults.
Both guards are mutation-verified:
- reverting
handler_tss.go→ fails on the widen assertion - stubbing out the
vault.Status != InitVaultcheck → fails on the active-vault assertion
go test -tags mocknet ./x/thorchain/... ./bifrost/signer/... green; scripts/lint.sh exits 0.
Rollout
Consensus-affecting, and the repo has no per-handler version gating — this must ride a coordinated minor release. This MR deliberately does not touch version or app/upgrades.go. No store migration: only vaults created after the upgrade height are affected. No proto or openapi regeneration.
Note this fixes future keygens only — the InitVaults from the stalled churn are cleared by the next retry regardless, and the two operators above still need to enable Solana.