# XMR Post-Churn Outbound Reliability - Root Cause, Impact & Fix Proposal | Repo | Branch analyzed | |---|---| | thornode/bifrost | `boonew/xmr-transparency-2026-05-09` (MR !4811, open) | | serai signer-sidecar | `zly/xmr-outbound-reliability-port` (https://gitlab.com/thorchain/devops/serai/-/merge_requests/15, open) | | alternate reviewed | `jpthor/monero` | ## Executive Summary XMR vault migrations stall permanently after a churn. The direct trigger is a signer-sidecar check that rejects peer-provided key images with a terminal HTTP 400. The structural cause: **because of Monero's privacy model, a validator that did not co-sign an XMR outbound cannot see it on-chain at all**, yet THORChain needs 2/3 of the *current active set* to attest every outbound. The bridge mechanism (importing the signer's key images into peer sidecars) fails on first use and never recovers. **How this regression happened:** the breaking check was added by serai MR 39 https://gitlab.com/thorchain/devops/serai/-/merge_requests/39 (merged via the consolidated audit MR 54 https://gitlab.com/thorchain/devops/serai/-/merge_requests/54) as a deliberate P1 security fix for audit finding https://gitlab.com/thorchain/devops/serai/-/work_items/320. Before it, the check was **too loose**. It only verified that key images appear in the spend tx, letting an authenticated caller mark *arbitrary* vault outputs spent. And precisely because it was loose, peer imports worked and earlier churn simulations passed. The audit fix swung to **too strict** (reject anything not already bound locally), which peers can never satisfy. The correct fix sits in between: **verified bind-on-missing**, restoring liveness while keeping the https://gitlab.com/thorchain/devops/serai/-/work_items/320 attack blocked. Other key findings: - Not a small-network quirk: on a mainnet-sized network, native observations mathematically **cannot** reach quorum (§4). The import path is the main path, not a fallback. - The issue is already tracked as thornode **#2896**, closed prematurely. Its claimed resolver (!4857, signer filtering) merged 2026-06-05, yet the simulation still failed 2026-07-01. Recommend reopening. - **No branch or MR on either repo currently attempts the sidecar fix** (§9). - Residual risk of the proposed interim fix is bounded, attributable griefing, not fund loss (§6). Trustless key-image proofs (jpthor-style) remain the long-term endgame. --- ## 1. How the issue was discovered The `xmr-post-churn` simulation (`make test-simulation-cluster TESTS=xmr-post-churn`) churns a mocknet from **1 node to 4 nodes** and waits for the XMR vault migration. It stalls reliably: - `vaults_migrating=true` forever, `xmr_active_vault_ready=false`, churn actors time out. - The first migrate **did** broadcast on Monero: txid `0dbc1db9…d37d`, memo `MIGRATE:1299`, old vault → new vault. - THORChain never processed the outbound: old vault not debited, pending migrate height not cleared. - After the stale-pending window, THORChain rescheduled a migrate for the **full original balance**. The wallet had already spent about half: ```text insufficient xmr balance for signing: available=122942108680000 required=245883917350000 ``` - Every non-signing peer logged the same sidecar error on import: ```text key image cannot be verified against tracked output ``` The 1→4 topology makes the failure deterministic: the source vault has one member, so exactly **one** validator can natively observe the outbound, while quorum needs **3 of 4**. The other three must co-attest via the import path, which was broken. --- ## 2. Background: why XMR outbounds are special Two different groups are involved in every XMR outbound: | Group | Who | Role | |---|---|---| | Signing committee | 2/3 threshold of the **source vault's** FROST membership | Signs and broadcasts the tx, learns the **key images** in the sign session | | Observation quorum | 2/3 of the **current active validator set** | Must attest before THORChain debits the vault | On transparent chains everyone can observe everything, so the split doesn't matter. On Monero, visibility depends on what you hold: | You hold | You can see | |---|---| | Vault view key (all members and registered peers) | **Incoming** funds (this is why the destination-vault credit worked) | | Key images (signing-session participants only) | **Outgoing** spends. To everyone else the tx looks like a stranger's | MR !4811's bridge: the signer's attestation carries the spent-output refs (key images). Non-signers import them into their local sidecar. Their own wallet then recognizes the on-chain spend, their own scanner emits an observation, and they co-attest. THORChain's voter merges the signer variant (with refs) and peer variants (without refs) into one quorum. --- ## 3. Root cause The peer import ends in `verify_spent_mark_key_image` (`serai …/scanner/tracker.rs`): if the tracked output has **no locally-bound key image → terminal `BadRequest`**. Key images are only ever bound during signing, so: - **Signers** pass the check, but they don't need the import. - **Non-signers** can never pass it. For them it's not verification, it's a guaranteed rejection. How the check evolved: | Era | Check | Security | Liveness | |---|---|---|---| | Before serai MR 39/MR 54 | key images ⊆ spend-tx inputs only | **Hole** (#320): caller could mark arbitrary vault outputs spent | Peer import worked, churn sims passed | | After !39/!54 (current develop) | must match an already-bound key image | Fixed | **Deadlocked** for peers | | Proposed | daemon-verified **bind-on-missing** + exact coverage | #320 stays blocked | Restored | The failure then cascades: 1. Sidecar returns 400 → Bifrost treats 400 as terminal (only 404/429/5xx retry) → peers never retry. 2. A one-shot flag (`spentRefsApplied`) latches *before* the import outcome is known → later attestations can't re-trigger it. 3. No peer observes → no 2/3 quorum → vault not debited, pending height not cleared. 4. Stale-pending expiry (10× signing period) reschedules the **full balance** → wallet refuses (`insufficient balance`) → permanent stall. A second path compounds it: refs for an output the peer hasn't *scanned yet* also return a terminal 400, though that condition is transient by nature. --- ## 4. Not a test-network quirk: the 100-node example 100-member monovault, signing committee = 2/3 = 67, quorum = 2/3 of active set. **Steady state (no churn):** only the 67 committee members can natively observe. Quorum needs 67 of 100, which is *exactly* borderline. One offline committee member means ordinary XMR outbounds never confirm without the import path. **Post-churn worst case** (33 rotated out, all from the committee): | Group | Count | Can natively attest? | |---|---|---| | Old committee members still active | 34 | ✅ | | Old non-committee members still active | 33 | ❌ view key only, no key images | | Newly churned-in nodes | 33 | ❌ | | **Quorum required** | **67** | - | 34 native attesters vs 67 required: **native observations cannot reach quorum in any churn scenario.** (Churned-out members still *co-sign* the migrate because membership is fixed at creation, but only active validators may observe.) **Mainnet consequence if the import path stays broken:** - every churn stalls the XMR migration - the retiring vault holds funds indefinitely and later churns back up - operators bonded to the retiring vault cannot exit - the full-balance retry wedges the signer against its own wallet A network-level liveness failure, triggered by every single churn. --- ## 5. What peers can and cannot verify After import, a peer verifies almost everything **itself**: - spend tx exists on-chain (daemon lookup) - claimed key images are inputs of that exact tx - tx pays the THORChain-scheduled destination/amount/memo (tx_key proof) - claim comes from an **active validator** that is a **source-vault member**, matching a **planned txout** (Bifrost authority checks) - after import, the peer's own scanner sees the tx in a block before attesting Exactly **one fact is borrowed on trust**: the mapping *"key image X belongs to vault output Y"*, which is cryptographically unverifiable from view-only data by Monero's design. Note the attacker model: key images are public once the tx broadcasts, so **any single malicious source-vault active validator** can attempt a false mapping, not just committee members. --- ## 6. What harm can the borrowed fact cause? With exact coverage enforced (claimed key-image set == the tx's full input set, distinct, correct count), only the *pairing* can be falsified: - **Permutation** (right outputs, shuffled pairing): spent-output *set* is correct → balances, solvency, input selection all stay right. Effectively harmless. - **Substitution** (real key images paired with *different* vault outputs): poisoned peers freeze live outputs, think spent ones are live → daemon rejections → **keysign failures until repaired**. Bounded because: - **No fund-loss path**: spending needs the FROST threshold, and a threshold-sized conspiracy could steal outright anyway. THORChain accounting stays fail-closed. - **Attributable**: the lie arrives in a signed attestation. - **Repairable**: rescan, plus letting thornode-committed quorum refs overwrite earlier peer-imported bindings. - **Ring membership nearly eliminates it**: the falsely-claimed output must sit among ~16 committee-chosen decoys out of hundreds of millions, which a lone attacker cannot steer. | Hardening level | Lone-attacker capability | |---|---| | bind-on-missing only | mark arbitrary outputs spent (real poisoning) | | + exact coverage, distinct key images | permutation or substitution (bounded, attributable griefing) | | + ring membership, conflict detection, committed-refs repair | permutation only (~harmless) | | key-image share proofs (jpthor-style DLEQ) | nothing (zero trust needed) | **Why not require a threshold of source-vault attesters before importing?** It can deadlock: in the worst case above only 34 committee members remain active, below a "2/3 of committee = 45" bar, and a 1-member vault must degrade to 1 anyway (that node *is* the vault and could take the funds outright). Agreement should be a **strengthener** (conflict detection: two authorized attesters disagreeing on refs for the same txid → hold), never a hard precondition. --- ## 7. Recommended fix Base: MR !4811 + serai MR 15 (https://gitlab.com/thorchain/devops/serai/-/merge_requests/15). (`jpthor/monero` has a broader model but **lacks the fail-closed migrate-credit guard**, crediting the destination vault unconditionally on a MIGRATE inbound, and would need that ported and the sim re-proven.) | # | Patch | Target | Content | |---|---|---|---| | 1 | **Sidecar import fix** (the liveness fix) | `devops/serai develop` (MR !15 as carrier) | Strengthen subset → **exact coverage**. **Bind-on-missing** via existing `set_output_key_images`. Change "not tracked" 400 → **404** (Bifrost already retries 404). Keep conflicts terminal, keep idempotency | | 2 | **Bifrost retry fix** | `thornode develop` (machinery already merged, coordinate with !4811) | Latch `spentRefsApplied` only on confirmed success. Add **conflict detection** for disagreeing authorized ref sets | | 3 | **Fast follow-up** (before production) | serai + bifrost | **Ring-membership check**. Treat **thornode-committed quorum refs as authoritative** (may overwrite earlier peer-imported bindings) | | 4 | **Endgame** | separate MR, own timeline | jpthor-style **DLEQ key-image share proofs**, deleting the borrowed fact entirely. Consensus-critical payload change (needs a version transition), so schedule deliberately, ideally before mainnet XMR value makes keysign-griefing attractive | Thornode side: keep the existing fail-closed migrate-credit guard (only in !4811) and durable pending-height handling. The "don't reschedule the full balance when the wallet reports less" safety net stays a separate, lower-priority item (needs XMR solvency data on thornode). --- ## 8. Acceptance criteria ```bash make test-simulation-cluster TESTS=xmr-post-churn ``` - first XMR migrate reaches `tx out processed chain=XMR` - old vault debited, new vault credited exactly once - pending migrate height removed, **no** second full-balance retry - `vaults_migrating=false`, signer wallets stay solvent Unit tests: - bind-on-missing success - unknown output → 404 → retry success - idempotent re-import - wrong or conflicting key image stays terminal - exact-coverage rejection of duplicate or short ref sets - Bifrost latch release on terminal failure --- ## 9. Related work (verified 2026-07-02) - **Origin of the breaking check:** audit finding `devops/serai#320`(https://gitlab.com/thorchain/devops/serai/-/work_items/320) (P1) → serai MR ! 39 (https://gitlab.com/thorchain/devops/serai/-/merge_requests/39), merged via consolidated audit MR !54. The audit attack is real (it matches the substitution analysis in §6). The fix simply overshot, see the era table in §3. - **Already tracked, closed prematurely:** thornode **#2896** ("reliability / 8: maintain migrate observation quorum across churn") describes this exact problem. Closed as "Resolved by !4857" (jail-aware signer filtering, merged 2026-06-05), which only *maximizes native attesters* by preferring active members for the committee, does nothing for non-signers, and demonstrably didn't fix the 2026-07-01 sim failure. **Recommend reopening** (or a successor under umbrella #2897). - **State of `develop` (both repos):** - thornode: has most Bifrost import machinery (authority checks, retry classification, catchup, latch) and the stale-pending-txouts fix (`cf4f504ea2`). It **lacks the migrate-credit guard** (only in !4811). - devops/serai: **still returns the terminal 400**. No branch or MR attempts the bind-on-missing/404 fix (MR !15 hardens adjacent paths only). Adjacent open MRs (complementary, none of them this fix): | MR | What it does | Relation | |---|---|---| | !4893 | Defer XMR solvency reports while sidecar balance is unreconciled (#2930) | Prevents false XMR solvency halts during the outbound window. Also *withholds* the wallet-balance signal a future full-balance-retry safety net would want | | !4894 | Cache signer result right after validated FROST result (#2853, partial) | Stops the *same node* re-building the same outbound on retry. Doesn't stop a *fresh* post-expiry migrate or help peer observation | | !4925 (draft) | Pin mocknet to `:develop` xmr-frost-signer image | Infra only. Sims run the #320-hardened sidecar, so the stall reproduces until Patch 1 lands | --- ## Appendix: key code references | What | Where | |---|---| | Terminal check causing the stall | `serai/xmr-frost-signer/src/scanner/tracker.rs` - `verify_spent_mark_key_image` | | Key images bound only during signing | `serai/xmr-frost-signer/src/sign/mod.rs` - `set_output_key_images` after FROST combine | | Peer import handler (daemon checks, tx_key proof) | `serai/xmr-frost-signer/src/scanner/mod.rs` - `mark_spent` | | Bifrost retry classification (400 terminal) | `bifrost/pkg/chainclients/monero/client.go` - `isRetryableSpentRefsSidecarErr` | | One-shot import latch | `bifrost/observer/attestation_handlers_impl.go` - `spentRefsApplied` | | Peer authority checks | `bifrost/observer/xmr_spent_refs.go` | | Variant merge at quorum (refs optional, tx_key exact) | `common/type_observed_tx.go` - `EqualsConsensus` | | Fail-closed migrate credit guard (!4811 only) | `x/thorchain/handler_observed_tx_helpers.go` - `shouldCreditObservedInboundVault` | | Stale pending migrate expiry (full-balance retry) | `x/thorchain/manager_network_current.go` (multiplier in `constants/constants.go`) | | Missing guard in alternate branch | `origin/jpthor/monero` - unconditional `vault.AddFunds` for `TxMigrate` |