test(simulation): divergent-node cluster scenario (one validator's local view perturbed)
🔎 Visual review: building it for5f0817b2; the link appears here.
What and why
In the bug retrospective, 15% of mainnet incidents were missed because they only show when validators' local views disagree (why_missed = multi-node-or-distributed-only, 28 fixes). Every validator in the mocknet cluster shares one daemon per chain, one token whitelist and a fresh token-metadata store, so none of these could appear:
- !4800 (merged): EVM solvency reports listed each node's locally discovered tokens, so nodes attested different coin sets.
- !5254 (merged) (bb5f627f): the BTC fee floor followed each daemon's relay fee. Core 29.1+ (100 sat/kvB) and older (1000) built different change outputs and split TSS.
- !5136 (merged) (BTC) and !5099 (merged) (ZEC): signers built transactions from their own daemon's UTXO view, so parties split.
- Same class, not covered here: !5340 (merged) and !4228 (merged) (XRP sequence), !5058 (merged) (ZEC expiry from the local tip), !4493 (merged) (TRON clock and reference block), !4920 (merged) (SOL solvency height), !3748 (merged) (BSC fee rolled out node by node).
This MR adds:
build/docker/docker-compose.divergence.yml, an overlay that perturbs one validator (cat) and leaves the other four at the default:bifrost-catgets its own BTC and LTC daemons, peered over P2P with the shared ones. These had to be added: the cluster shares one daemon per chain across all five bifrosts.- BTC uses Core 29.1's 100 sat/kvB relay fee, the exact !5254 (merged) case.
- LTC uses a 10x stricter one, set by
DIVERGENCE_LTC_MINRELAYTXFEE; set it empty to keep the default policy. - Each own daemon also brings its own mempool, UTXO view and wallet imports.
bifrost-cat's BASE token whitelist lacks the simulation TKN token. Its token-metadata store never learns a token its peers know, as after a rolling whitelist upgrade or a different discovery history. Against pre-!4800 code this gives cat a different solvency coin set.
- An opt-in
divergencestage (actors/features/divergent_node.go, run afterchurn):- setup: fails fast unless the divergent daemons are reachable, peered and synced, with a different BTC relay fee.
- activation: waits up to ten minutes for cat to churn in. The
churnstage adds only three of the four standby validators. With equal bonds, ready nodes join in node-address order, and cat's address sorts last, so cat joins on the next churn. If cat never becomes eligible, the error quotes THORNode's preflight reason and calls it a divergence finding. - freeze: halts churning, then records cat's Asgard vaults. Churning resumes at the end of the stage.
- traffic: two rounds of seven concurrent swaps across BTC, LTC and BASE.TKN, so the same vault has unconfirmed change to disagree about.
- verify, in order:
- The BTC, LTC and BASE outbound queue must drain.
- For two minutes, the stage reads the solvency voters from the store (
abci_query /store/thorchain/subspace, since there is no endpoint for them). Bifrost reports a vault only when it finds it insolvent, so healthy vaults have no voters. The stage fails if cat attests more insolvency reports for its own vaults that no supermajority confirmed than the noisiest peer, plus two. The error names the divergent local view. - No validator may be jailed.
- A
KeysignFailureswatcher, started only with this stage, that fails the run on any agreedtss_keysign_failureand names the blamed validators. make test-simulation-cluster-divergence(local),-native(CI) andstop-mocknet-cluster-divergence. Once the sim passes, the target grepsbifrost-cat's log for the refused BASE token, because the sim cannot read bifrost's config.- A manual, allow-failure CI job,
test-simulation-divergence, which extendstest-simulation.
Was it run end to end?
Two CI runs so far, neither green. Both failures were in the harness, now fixed. A third run is pending.
- Job 18071366264 failed in setup: cat was still standby after the
churnstage.- The churn sets
NUMBEROFNEWNODESPERCHURN=3, and cat's address sorts last among four equal-bond standby nodes. - Fix: wait for the next churn, then freeze the vault set.
- The churn sets
- Job 18073009385 got further: cat churned in (setup 9m46s), churning halted, and all 14 swaps completed with their outbounds (BTC, LTC, BASE.TKN). The outbound queue drained at once, and no keysign failure was agreed (the watcher stayed quiet).
- Verify then timed out, waiting for three BTC solvency reports to reach consensus on cat's vaults.
- No report ever would. UTXO and EVM bifrosts send a solvency report only for a vault they find insolvent, or an all-clear while the chain is halted.
- The eight agreed voters on other vaults were the old vault's insolvencies during migration. bifrost-cat logged no BTC or LTC insolvency once it was active.
- Fix: measure divergence as unconfirmed insolvency reports, as described above.
What the second run shows for the divergence itself:
- BTC: converged. Cat's 100 sat/kvB daemon did not change anything, as expected since bb5f627f pinned the floor.
- LTC: no split. Cat built the same four LTC transactions as fox (same totals and gas). Mocknet LTC outbounds pay 78,819 to 210,000 sats, far above cat's 10,000-sat floor, so the default perturbation never moved the fee.
DIVERGENCE_LTC_MINRELAYTXFEE=0.003(a 300,000-sat floor) is the setting that exercises the daemon-derived LTC floor at mocknet fee levels. Against mainnet's few-hundred-sat LTC fees, the default 10x gap would already apply. - BASE: the BASE.TKN swaps both ways completed. bifrost-cat refused TKN metadata 27 times ("token not whitelisted"), so the whitelist perturbation was in effect. The Make target's own check runs only after a passing sim, so it did not run.
- Keysign blame: none agreed.
- Jailing: not reached in that run.
This host could not run the cluster locally. Another session's mocknet containers share the compose project and hold its host ports. To run it:
make test-simulation-cluster-divergence PARALLELISM=8 CI=1
make stop-mocknet-cluster-divergenceor start the manual test-simulation-divergence job on this MR's pipeline.
Also verified:
- Compose config renders.
- The exact
bitcoin-catandlitecoin-catcommands peer, sync and takecreatewalletandimportaddress, run in isolated containers. - Unit tests for every new piece.
scripts/test-simulation-unit.sh,make lint-ci,make generate(clean) andglab ci lint.
Third run: passed (job 18075783119, tip 5f0817b2)
Job 18075783119 succeeded end to end, as did pipeline 3074916332.
- Setup: cat churned in after 9m51s, and the stage froze churning while cat was a vault member.
- Traffic: two rounds of BTC, LTC and BASE.TKN swaps; the outbound queue drained.
- Verify: passed after its 2-minute solvency watch.
- cat filed no unconfirmed insolvency reports for its own vaults, on BTC, LTC or BASE.
- No keysign failure was agreed.
- Whitelist check: after the run, the Make target confirmed that bifrost-cat refused the BASE token ('token not whitelisted').
- One warning: cat filed one unconfirmed LTC insolvency report, for a vault it is not a member of. That is consistent with the fresh-wallet finding below: a pre-existing vault its own daemon never imported.
With the default perturbation, divergent relay-fee policy, mempool and token list do not split signing or solvency for a vault's own members. The fresh-wallet import bug is the one real divergence this scenario observed.
Observed finding: bifrost never imports vaults that predate it into a fresh daemon wallet
In both runs, before it churned in, bifrost-cat reported BTC and LTC solvency for the genesis vault with wallet_amount 0 against an Asgard amount of 2.5. In run 1 that was 47 BTC and 11 LTC "insolvency detected" reports. The other four bifrosts share the funded daemon and saw the balance.
Cause, in cmd/bifrost/main.go:
pubkeyMgr.Start()fetches the existing vaults and fires the new-pubkey callbacks beforeLoadChainsregistersutxoClient.RegisterPublicKey.- So vaults that exist when bifrost first starts are never
importaddressed into its daemon wallet. bifrost has no rescan import either. - Its
listunspentfor those vaults stays empty, andGetAccountand solvency read 0.
On mainnet this hits a node whose daemon wallet is fresh, such as a new validator or a rebuilt daemon:
- It attests insolvency for every vault that existed before its bifrost started, for example a retiring vault during its first churn, until those UTXOs are spent.
- Its
GetAccountfor those vaults is wrong.
The shared mocknet daemon hid this, because the genesis node imported the vault for everyone.
The stage asserts only on vaults cat is a member of, which are created after cat's bifrost started. It logs cat's reports on other vaults as this finding. A fix needs a human decision and its own MR, for example:
- import the known pubkeys once the UTXO clients have registered;
- or import with rescan.
From reading the code (not observed)
- The LTC, DOGE and BCH fee floor is still the daemon's
relayfee;updateNetworkInfopins only BTC.applyFeeLimitsuses it as an absolute fee floor. A node whose daemon has a higher relay fee than the outbound's fee raises the fee and signs a different message: the !5254 (merged) split on those chains. The second run did not trigger it, because mocknet fees are high. UseDIVERGENCE_LTC_MINRELAYTXFEE=0.003to test it. If it splits, deciding whether to pin those floors is a bifrost signing change for a human. TokenManager.GetTokenMetastores the default decimals permanently when thedecimals()call fails on first discovery. A node that hit an RPC error then converts a 6-decimal token with 18 decimals in signing and solvency. Mocknet tokens are all 18-decimal, so this scenario cannot show it.
Not covered
- Restarting bifrost mid-keysign on one node is not built. It needs container control from the sim, or a Make-level
docker restart bifrost-cattimed to a keysign. - Only one validator is perturbed, and the four default ones keep quorum. A bug needing many divergent nodes shows here only as cat's unconfirmed reports or blame, not as failed consensus.
- No per-node clock skew, scanner lag, or XRP/ZEC/TRON local-state divergence.
- The stage cannot see whether cat was in each keysign party.
Follow-ups
- Get a green run, then run once with
DIVERGENCE_LTC_MINRELAYTXFEE=0.003to test the daemon-derived LTC floor. - Decide on the fresh-wallet import finding above.
- Add a bifrost restart mid-keysign, and divergent ZEC (own
zcashd, !5099 (merged)/!5058 (merged)) and XRP (ownrippledview, !5340 (merged)) daemons. - Once green repeatedly, consider adding
divergenceto the regular cluster job.