# XMR Monovault Keygen Mass-Jailing - Root Cause, Repro & Fix Plan ## Summary The XMR monovault FROST keygen mass-jails members on churn (live chainnet incident, ~79/95-style jailing via `judgeLateSigner` on a retry-round success). Churn is currently held by the network-wide bifrost keygen skip (!4932), with the `HaltMonovaultKeygen` mimir (!4919) as the cleaner 3.20 lever, but both only *avoid running* the keygen. Root cause is **per-round timing skew, not malice or node failure**: a member whose round timer expires moments before the last peer message abandons a DKG it had effectively completed. Three compounding defects then turn one slow message into a full-term jail sentence. All three are reproduced deterministically by unit tests and a 4-node mocknet cluster simulation. ## The three defects ### 1. Different-ID trap (on-chain, by design - wrong at scale) A timed-out member submits a keygen *failure with blame*. `getTssID` hashes the blame into the msg ID, so the failure vote's tss ID differs from the success ID the finishers report. At the first block where the success reaches 2/3 of members, `judgeLateSigner` (`x/thorchain/handler_tss.go`) jails **every member that has not signed the success ID** - which a failure submitter can never do. Release requires a same-ID signature, re-submitting the failure is silently dropped as a duplicate vote. Jail sticks for the full `JailTimeKeygen` (4320 blocks), plus `FailKeygenSlashPoints`. ### 2. Broken self-heal (on-chain, plain bug) `judgeLateSigner` receives the voter **by value** and persists `MajorityConsensusBlockHeight` itself - but the vault-finalize path that runs immediately after calls `SetTssVoter` with the handler's **stale copy**, zeroing the field again. XMR monovault keygen always finalizes on the consensus-reaching message, so the designed late-signer release path never fires: an honest member whose *success* report lands one block after consensus is jailed and never released, and every subsequent late report re-enters the "first consensus" branch and **re-jails all remaining absentees** with a fresh release height. (Asgard keygens without check-sig quorum dodge this only via an early return.) ### 3. Unreachable healing path (bifrost) Bifrost already contains the correct recovery: after a failed local execution, `frostKeygen` waits for peers' result broadcasts and adopts a quorum success (`frost_keygen.go`). It was dead code - the wait used `quorum = n`, while a node whose own execution failed can observe at most **n−1** peer successes. Additionally, a peer that fully participated in rounds 1–2 and misses only the round-3 window is blamed exactly like a node that never showed up. Additionally, the result waiter itself was fragile: `WaitForResult` returned a terminal failure on the first malformed or unauthorized DKG result message. The keygen msg ID is derivable by any connected peer, so one junk message arriving before the genuine quorum could veto the adoption path and every non-active waiter - and the terminal response carried blame derived from unauthenticated gossip noise, which could end up in the node's on-chain failure vote. ## Why every large keygen risks this FROST DKG requires all n members to complete every round, with a full transcript verification matrix. Round messages at n=95 are ~n² sends against a flat per-round timer (`frost_keygen_timeout`, 60s floor). Whoever crosses the timer last becomes either a blamer or the blamed, submission spread does the rest via defects 1–2. The split is structural: it does not require any node to be broken. ## Reproduction (deterministic) - Unit: `TestXMRMonovaultKeygenFailureSubmitterStaysJailedAfterSuccessConsensus` (x/thorchain - pins defects 1+2, including the re-jail cascade) and `TestFrostKeygenExecuteBlamesAlmostCompleteFinisherOnLastRoundTimeout` (bifrost - pins defect 3). - Cluster: `make test-simulation-keygen-jail` - netem latency on the bifrost p2p port (so DKG transcript verifications lag round payloads by one hop) plus a mocknet-build-only round-3 wait override on one bifrost, placing its timer inside that one-hop window. Reproduced at 4 nodes in ~4m30s: new XMR vault created by the finishers, participating members jailed with "failed to vote keygen in time" - including a *second* member that merely reported success late (defect 2 live). ## Fix plan | # | What | Where | Status | |---|---|---|---| | 1 | **Quorum-success adoption**: lower the post-failure result wait to the on-chain supermajority (ceil 2n/3) and gate adoption on a keyshare proof - sidecar finalize validates the local DKG session derives exactly the adopted group key/address and returns the keyshare, producing the same success tss ID the finishers report. Also hardens the result waiter: invalid result messages are logged and skipped instead of ending the wait (mirroring the keysign waiter), and vote accounting is last-write-wins per sender. Fixes defects 1+3 at the source. | bifrost (non-consensus, 3.19.x-class) | MR !4934 | | 2 | **Persist `MajorityConsensusBlockHeight` correctly** so the late-signer release works and the re-jail cascade stops. Fixes defect 2. | x/thorchain (versioned) | follow-up MR | | 3 | Optional hardening: grace window before keygen jail (or slash-points-only for `XMRMonovaultKeygen`). | x/thorchain (versioned) | follow-up | ## Acceptance criteria - `make test-simulation-keygen-jail` with fixes 1+2: same fault injection, vault created, **nobody stays jailed**. - The keygen skip (!4932) can be removed and churn re-enabled with monovault keygen on. ## Related - !4932 (network-wide bifrost keygen skip - the current workaround this issue unblocks removing) - !4919 (`HaltMonovaultKeygen` mimir - operational lever, 3.20) - Orthogonal to #3090 / !4931 / serai!15 (XMR *outbound observation* reliability)