fix(xmr / bifrost): adopt quorum FROST keygen success after a recoverable local failure (keygen mass-jailing)

Scope / status: This MR stops the mass-jailing, but it does not fully resolve the incident on its own. With this MR alone, every keygen/churn still leaves exactly one active validator jailed for ~7h (JailTimeKeygen, 4320 blocks) with ~720 unrestored slash points - the one member whose success report lands after on-chain consensus. No fund risk, no liveness risk. Fully closing it needs a small consensus-lane x/thorchain change (defect 2 below), so until that lands the keygen skip (!4932 (merged)) should stay in place.

Problem

The XMR monovault FROST keygen mass-jails members at churn (live chainnet incident, ~79/95-style jailing. Churn is currently held by the network-wide keygen skip !4932 (merged)). Root cause is per-round timing skew, not malice: a member whose round timer expires moments before the last peer message abandons a DKG it had effectively completed, and three defects turn that into permanent jail:

  1. Different-ID trap - the member submits a keygen failure with blame, and getTssID hashes the blame, so the failure vote's tss ID differs from the success the finishers report. When they reach 2/3 consensus, judgeLateSigner jails every member that has not signed the success ID - which a failure submitter can never do.
  2. Broken self-heal (on-chain bug, not fixed here) - judgeLateSigner receives the voter by value and persists MajorityConsensusBlockHeight itself, but the vault-finalize path immediately after calls SetTssVoter with the handler's stale copy, zeroing it: late same-ID signers are never released, and each late report re-jails all remaining absentees.
  3. Unreachable healing path - bifrost already waits for peers' result broadcasts after a failed local execution and adopts a quorum success, but the wait used quorum = n, which a node whose own execution failed can never see (at most n-1 peer successes exist).

What this MR does (bifrost + tests only, no state-machine changes)

Fix (defects 1+3): lower the adoption quorum to the on-chain supermajority (ceil 2n/3, mirroring HasSuperMajority) - adopt exactly when THORChain will finalize the vault - and gate adoption on a keyshare proof: AdoptQuorumSuccess finalizes the local sidecar DKG session against the adopted group key and address. The sidecar validates the stored session material derives exactly that vault and returns the keyshare backup, so the adopted response carries the same success tss ID the finishers report. A member whose session did not complete fails the proof and keeps its failure - a node without a keyshare can never report success.

Repro unit tests pinning the full mechanism on both sides (TestXMRMonovaultKeygenFailureSubmitterStaysJailedAfterSuccessConsensus, TestFrostKeygenExecuteBlamesAlmostCompleteFinisherOnLastRoundTimeout - the latter now also covers the adoption healing).

Deterministic cluster repro: make test-simulation-keygen-jail - netem latency on the bifrost p2p port plus a mocknet-build-only round-3 wait override on one bifrost (//go:build mocknet. Production builds compile a constant-zero stub). Reproduced the incident at 4 nodes in ~4m30s: vault created by the finishers, participating members jailed.

Result-waiter robustness (prerequisite for the above): WaitForResult previously returned a terminal failure on the first malformed or unauthorized DKG result message - and the keygen msg ID is derivable by any connected peer, so a single junk message could veto the adoption path, kill every non-active waiter, and inject gossip-derived blame into this node's on-chain failure vote. Align it with the keysign result waiter: invalid messages (unparseable payload, empty sender, non-active sender, peer-binding mismatch, structurally invalid result) are logged and skipped. The wait ends only on quorum or timeout. Vote accounting is now last-write-wins per sender, so a member cannot hold votes in two result buckets at once (this also bounds the bucket map by participant count). Both properties are pinned by tests that fail on the previous code.

How to run the xmr keygen jail simulation test

Use the dedicated target:

make test-simulation-keygen-jail

It runs the stage:

STAGES=seed,bootstrap,xmr-keygen-jail

and also applies the required netem latency/fault-injection setup first.

If you need to tune the repro timing:

XMR_KEYGEN_JAIL_ROUND3_TIMEOUT_MS=3600 \
XMR_KEYGEN_JAIL_NETEM_DELAY_MS=2500 \
make test-simulation-keygen-jail

Those are already the current defaults.

What this MR does NOT fix (residual: one jailed node per churn)

This MR removes the mass-jailing, but defect 2 is not fixed here, and on its own it leaves exactly one jailed active validator per keygen/churn: the first member whose success report arrives strictly after the block that crosses on-chain consensus. Because judgeLateSigner persists MajorityConsensusBlockHeight on a by-value voter that the vault-finalize path then re-writes as zero, that first post-consensus reporter is jailed and never released (~7h jail, ~720 unrestored slash points). This is structural and independent of validator-set size - it is why the xmr-keygen-jail sim still asserts a jailed victim at 4-node timing (adoption costs the p2p round-trips the 3 finishers don't pay). There is no fund or liveness risk, but on a small active set even one jailing can threaten quorum.

What remains (follow-up, consensus lane)

Defect 2 needs a small x/thorchain change to persist MajorityConsensusBlockHeight correctly (versioned/consensus-breaking release). When it lands, the residual goes to zero and the xmr-keygen-jail sim assertion flips from "victim jailed" to "no victim". An optional grace window before jailing is a further hardening. Until then, keep !4932 (merged).

  • Part of #3092 (closed) (keygen mass-jailing tracking issue) - not a full close, the consensus-lane defect-2 fix is still required.
  • Unblocks removing !4932 (merged) (network-wide keygen skip) only once defect 2 lands, complements !4919 (HaltMonovaultKeygen mimir): both are levers to not run the keygen. This MR makes running it survivable at scale, minus the one-node residual.
  • Orthogonal to !4931 (merged) / thorchain/devops/serai!15 (merged) (XMR outbound observation reliability): no dependency, any merge order.
Edited by ZlyDevMaya

Merge request reports

Loading