fix(bifrost): make XMR spent-refs import outcome-aware so peer observation recovers

ZlyDevMayarequested to merge
zly/xmr-outbound-reliability-develop into develop

Closes #3090 (closed)

Problem

XMR vault migrations can get stuck after a churn (#2896 (closed), #3090 (closed), umbrella #2897).

Monero transactions are private. A node that did not help sign an outbound usually cannot find that transaction on-chain by itself. However, THORChain still needs observations from two-thirds of the active nodes.

The existing recovery flow has several problems:

  1. Another node may try to import the spent-output refs before its scanner knows about the original outputs.
  2. Bifrost previously marked the import as completed before the sidecar confirmed success.
  3. If the import failed, later attestations did not retry it.
  4. Without enough observations, THORChain does not debit the old vault or complete its migration.
  5. Recovery may then try to send the original vault balance again, even though some funds were already spent.

This failure was reproduced with:

make test-simulation-cluster TESTS=xmr-post-churn

What this MR changes

Retry failed spent-ref imports

  • Mark an import as completed only after the sidecar confirms success.
  • Return the final result through a callback.
  • Retry HTTP 404 responses and partial responses with missing outputs.
  • Stop retrying after a limited number of attempts.
  • Release temporary reservations when an import fails.
  • Prevent retry goroutines from racing with client startup or shutdown.
  • Reject duplicate key images and conflicting ref sets.
  • Restore spent-ref imports from signer observations.

The number of imports running at the same time is limited. If all available slots are busy, the import fails immediately and is retried later by reconciliation.

Require approval from the old vault

Spent refs change the local XMR wallet state, so they must not be accepted from a single untrusted peer.

Before importing refs, Bifrost now checks that:

  • The observation has a valid signature.
  • The signer matches the peer that sent it.
  • The signer was a member of the old source vault.
  • Enough source-vault members signed the same observation.
  • The observation matches the planned outbound.
  • The ref set does not conflict with another approved ref set.

These signatures only approve the local spent-ref import. They do not count toward the active-node observation quorum sent to THORChain.

Allow churned-out members to send certificates

Members of the old vault may have the information needed to recover an outbound even though they are no longer active validators.

This MR adds a separate certificate protocol for that information. It includes:

  • Message size limits and read timeouts.
  • Per-peer and global rate limits.
  • Limits on concurrent Thornode RPC calls.
  • Short-lived caches for validated vault and txout data.
  • Limits on stored certificates.
  • Retry backoff.
  • Bounded certificate broadcasts.
  • Background processing so slow XMR certificate work does not block the shared observation queue.

Certificates from old members can help approve a local import, but they never count as active-validator observations.

Include verified refs in peer observations

A signer observation contains spent refs. Previously, a peer that imported those refs could observe the transaction but submit it without the refs. THORChain would treat these as different observation versions.

After a successful import, the sidecar returns the verified refs with the outgoing transfer. Bifrost now includes them in the peer observation.

Bifrost only includes refs when:

  • The transaction is an XMR outbound from the expected vault.
  • Every ref belongs to the observed transaction.
  • The complete ref set passes validation.

Refs are never added to inbound observations.

Compare XMR addresses correctly

XMR addresses are case-sensitive.

This MR adds common.AddressEquals:

  • XMR addresses use exact comparison.
  • Other chains keep the existing case-insensitive behavior.

Tests cover both cases.

Update the XMR sidecar

This MR uses:

registry.gitlab.com/thorchain/devops/serai/xmr-frost-signer:16f06497@sha256:5224d4fc192ead39ab313b7f96282994204e895c0f06c8af65ed9dc7abe599f1

The matching sidecar changes are merged in devops/serai!15.

Scope

This is a Bifrost-side reliability fix.

It does not change the THORChain state machine or consensus rules. Nothing under x/ or proto/ is changed.

The Make observer deck and errata processing durability issue is intentionally not resolved in this MR. With around 100 nodes able to observe XMR transactions, one or two nodes losing local observation state should not create a chain-wide liveness problem. This broader durability improvement is left for later follow-up work.

Rollout

XMR observations have not been enabled yet. They will remain disabled until all nodes are upgraded.

Because of this:

  • There are no old XMR observations to support.
  • There will be no mixed old/new observation window.
  • No compatibility code for unreleased formats is needed.

Related

  • #2896 (closed) describes the original post-churn problem.
  • !4857 (merged) improves signer selection but does not fix observation recovery after signing.
  • !4811 adds XMR outbound transparency information and complements this MR.
Edited by ZlyDevMaya

Merge request reports