fix(observer): S13 - prevent XMR quarantine from stalling shared errata processing

Prevent slow XMR quarantine from stalling Observer's shared scanner errata consumer. Move broad reconciliation into one owned maintenance loop so unrelated chains' errata can continue to be durably accepted while XMR cleanup retries.

Series context

This is S13 of the focused replacement MRs for the superseded XMR MR !4983 (closed). The complete series plan is tracked there.

  • Scope: isolate shared errata processing from XMR quarantine latency and manage the reconciliation lifecycle.
  • Merged dependency: !5068 (merged) / S12 - terminal errata outcomes and retries, including the exact-hash broadcast fence and pending-REVERTED admission block, built on S09/S10’s ordering and mined-height identity.
  • Already delivered: !5124 (merged) / S13a - standalone startup isolation and signing readiness.
  • Related work: !5067 / S11 - spent-reference recovery and its shared errata admission guards.
  • Follow-ups: P05 - storage indexing, S15 - state-sync cancellation.

This MR targets develop, which now includes !5068 (merged) / S12. Its remaining runtime isolation scope is separate from the startup/readiness NEED item delivered by S13a.

Changes

  • Extend S13a's one-shot startup recovery worker into one owned maintenance loop.
  • On nodes with an XMR client, reconcile incomplete records immediately after any required completed-history restoration succeeds, then retry at the existing block-time cadence: six seconds normally, one second in mocknet.
  • Move broad reconciliation out of both committed-event callbacks and the shared errata consumer's retry path.
  • Retry incomplete quarantine across failures and restart. Preserve durable safety history and completed-tombstone quarantine restoration at startup.
  • Stop maintenance between records, let non-XMR clients stop, and join maintenance before stopping the XMR client or closing Observer storage, leaving unfinished records durable for restart.
  • Replace the sequential-callback test with a regression that drives the real shared errata consumer with BTC blocks while XMR quarantine is blocked.

The production changes are confined to bifrost/observer/observe.go. Signing readiness and the durable broadcast/admission protections are inherited from S13a and S12. This MR adds no consensus-rule, wire-format or persisted-schema changes.

Known limitations and follow-ups

  • After a REVERTED receipt, TxID-wide deck cleanup and scanner quarantine can wait up to one block time before the next maintenance pass: six seconds on mainnet, one second in mocknet. S12's exact-hash fence and pending-REVERTED admission block cover that scheduling window. Slow or failing operations can extend completion time.
  • The exact broadcast fence shares the signed-payload store mutex with history scans, so delivery can still wait on store contention.
  • Maintenance retains a full tombstone scan per block time. This scan already exists in S12's retry path - this MR moves it to nodes with an XMR client and removes callback-triggered scans.
  • In-flight storage or REST operations are not cancelled. No hard shutdown-latency guarantee is introduced.
  • S12’s deferred late-outbound and common queued-swap counterexamples remain opt-in in the inherited base.
  • While startup quarantine restoration is failing, reconciliation of newer XMR receipts also waits. XMR remains unready, and the existing broadcast and admission protections remain in force.

Verification

  • The shared-consumer regression fails on S12 alone: the first BTC block is durably accepted, but the second cannot enter the consumer within two seconds while XMR quarantine is blocked. It passes with S13 in normal and mocknet builds.
  • Full mocknet Observer and Monero packages passed.
  • Focused race tests passed with -race -count=3.
Edited by ZlyDevMaya

Merge request reports

Loading
Loading