## Summary Implement automatic failover logic so that when a chain data provider goes down or falls behind the chain tip, Bifrost automatically switches to a healthy backup provider. This ensures continuous chain observation and signing even when individual providers experience issues. Depends on: #2827 (Support multiple chain data providers) ## Motivation Running a single data source per chain is a single point of failure. If a validator's local daemon crashes, falls behind, or becomes unresponsive, that validator stops observing and signing transactions until the daemon recovers. With multiple providers configured, Bifrost should automatically detect degradation and failover to a healthy provider. ## Failure Modes to Detect 1. **Provider is down** — connection refused, timeouts, HTTP 5xx errors 2. **Provider is behind chain tip** — block height is stale compared to other providers or known chain tip 3. **Provider is returning errors** — repeated RPC errors, rate limiting (HTTP 429), auth failures 4. **Provider is slow** — response latency exceeds acceptable thresholds 5. **Provider returns inconsistent data** — block hash mismatches or missing transactions (data integrity) ## Proposed Design ### Configuration ```yaml # Example config per chain bifrost: chains: btc: providers: - name: "local-daemon" rpc_host: "http://localhost:8332" priority: 1 # lower = preferred type: "bitcoin-rpc" - name: "quicknode" rpc_host: "https://xxx.quiknode.pro/abc123" priority: 2 type: "bitcoin-rpc" - name: "blockchair" rpc_host: "https://api.blockchair.com" priority: 3 type: "blockchair-rest" # uses adapter failover: health_check_interval: 10s # how often to check provider health max_block_lag: 3 # max blocks behind tip before considered stale error_threshold: 5 # consecutive errors before marking unhealthy error_window: 60s # time window for error counting recovery_check_interval: 30s # how often to re-check failed providers min_recovery_checks: 3 # consecutive healthy checks before restoring ``` ### Health Check System Each provider is continuously monitored with periodic health checks: - **Liveness check**: Can we connect and get a response? - **Block height check**: Query latest block height, compare against best known tip across all providers - **Latency tracking**: Rolling average of response times - **Error rate tracking**: Sliding window of error counts ### Provider States ``` ┌──────────────────┐ │ HEALTHY │ │ (serving traffic)│ └────────┬─────────┘ │ error_threshold exceeded │ OR block_lag > max_block_lag ▼ ┌──────────────────┐ │ UNHEALTHY │ │ (failover target)│ └────────┬─────────┘ │ recovery_check_interval ▼ ┌──────────────────┐ ┌───── │ RECOVERING │ │ │ (being re-checked)│ │ └────────┬─────────┘ │ │ min_recovery_checks passed │ ▼ │ ┌──────────────────┐ │ │ HEALTHY │ │ └──────────────────┘ │ │ recovery check failed └──────► back to UNHEALTHY ``` ### Failover Logic 1. On startup, all configured providers are health-checked and the highest-priority healthy provider is selected as **active** 2. A background goroutine runs health checks on **all** providers (active and inactive) at `health_check_interval` 3. When the active provider becomes unhealthy: - Immediately switch to the next healthy provider by priority - Log a warning with the reason for failover - Emit a Telemetry metric for alerting 4. When a higher-priority provider recovers: - After passing `min_recovery_checks` consecutive health checks, mark it healthy - Optionally switch back to it (configurable: `prefer_higher_priority: true`) 5. If **all** providers are unhealthy: - Log an error - Continue retrying all providers at `recovery_check_interval` - Emit a critical Telemetry metric ### Block Tip Consensus To detect stale providers, maintain a "best known tip" per chain: - Query block height from all configured providers on each health check cycle - The highest reported block height (seen by 2+ providers if multiple are configured) is the consensus tip - Any provider more than `max_block_lag` blocks behind the consensus tip is marked stale/unhealthy - If only one provider is configured, use the chain scanner's last observed height as the baseline ### Telemetry & Observability Expose the following metrics: - `bifrost_provider_status{chain, provider, status}` — current state of each provider (healthy/unhealthy/recovering) - `bifrost_provider_block_height{chain, provider}` — latest block height per provider - `bifrost_provider_latency_ms{chain, provider}` — response latency per provider - `bifrost_provider_errors_total{chain, provider}` — error count per provider - `bifrost_provider_failover_total{chain}` — number of failover events per chain - `bifrost_provider_active{chain, provider}` — which provider is currently active ### API / Admin Endpoints - `GET /thorchain/providers` — list all providers and their health status per chain - `GET /thorchain/providers/{chain}` — detailed health info for a specific chain's providers ## Tasks - [ ] Design `ProviderManager` interface that wraps provider selection and failover - [ ] Implement health check goroutine with liveness, block height, and latency checks - [ ] Implement provider state machine (healthy → unhealthy → recovering → healthy) - [ ] Implement block tip consensus logic across multiple providers - [ ] Implement automatic failover on active provider degradation - [ ] Implement automatic failback to higher-priority recovered providers - [ ] Add failover configuration schema to `config.yaml` - [ ] Add Telemetry metrics for provider health and failover events - [ ] Add admin API endpoints for provider status visibility - [ ] Integration tests with simulated provider failures - [ ] Documentation for validators on configuring failover