node-status: fix false-healthy reporting, add bifrost scanner columns, issue probes concurrently
Summary
make status renders several failure modes as success — most seriously, a chain whose endpoint answered nothing at all shows 100.000%. The endpoint fallbacks added for BIFROST_CHAINS_*_RPC_HOST make this easy to hit, because those endpoints are full URLs rather than the bare host:port of an in-cluster daemon. The table also says nothing about whether bifrost itself is keeping up with the daemons it observes, and the probes were issued one at a time, so no two rows described the same instant.
Four commits, each reviewable on its own:
- Accuracy — every path that rendered a failure as health, plus the table restructure
- Concurrency — issue the independent probes together
- JSON-RPC error bodies — treat an error response as a missing height, not the string
null - TRON probe gating — a fullnode no longer queries TRON endpoints for a row it does not print
The table
SYNC and BEHIND are now a pair reported per component, ordered so each column is measured against the one to its right: bifrost's block scanner trails the daemon, the daemon trails the TIP reference. Negative BEHIND means behind in both.
BIFROST DAEMON
CHAIN SYNC BEHIND SYNC BEHIND TIP
THOR - - 100.000% 0 27,326,852
BTC 100.000% 0 100.000% 0 961,390
ETH 100.000% -1 100.000% 0 25,701,482
ETH (beacon slot) - - - - -
GAIA 100.000% -1 100.000% 0 32,387,127
BSC 100.000% -13 100.000% 0 114,502,014
...The BIFROST pair comes from /status/scanner and reports what the daemon columns structurally cannot: a daemon sitting exactly at tip whose scanner is stalled or catching up previously read 100.000% 0, and that lag is what accrues observation slash points. The cells distinguish the states bifrost reports — a lag in blocks, Error for an unreadable height, missing for an enabled chain bifrost holds no client for (its endpoint was unreachable when bifrost started, which becomes invisible once the endpoint recovers), and an asterisk plus footnote when the scanner reports itself unhealthy. THOR has no scanner height to report and a fullnode has no bifrost, so both read -.
The endpoint is requested once and cached; every row parses that one response.
Two details of reading that endpoint are worth noting. Bifrost reports -1 as its could-not-read sentinel, but scanner_height_diff also takes -1 in ordinary operation: the handler fetches the chain height live while the scanner height comes from local storage, so a scanner one block ahead of the tip GetHeight() happened to return — ordinary skew when the endpoint is load-balanced — reports a diff of exactly -1. The sentinel is therefore tested on the heights, and BEHIND is computed from those two heights in the daemon column's sign convention rather than from the reported diff. All three fields are selected in one pass with @tsv, since a separate jq invocation per field is a process apiece for every row.
Accuracy fixes
calc_progressreturned100.000%whenever both heights were empty, because"" = ""compares equal, so an unreachable endpoint rendered as fully synced. A zero reference height renderedinf%. Heights are now accepted only as plain non-negative integers.- A JSON-RPC error body — a rate limit, a revoked key — is served with HTTP 200, so
curl --failpasses it through, andjq -rprints the literal stringnullfor the absent field. That compared equal exactly as the empty strings did. All heights are now extracted through a helper that yields nothing for a missing field, which also fixes SOL, whose.result // 0turned a dead endpoint into height0, and the EVM chains, which fednulltoprintf %.0fand got a confident0plus an error on stderr. - The UTXO probes prefixed
http://thorchain:password@onto the endpoint for the in-cluster daemons' credentials. Applied to a URL that already carries a scheme the request is malformed and returns nothing, which the above then rendered as 100%. The prefix now applies only to scheme-less endpoints. - Several chains substituted a sync height of
0when their endpoint returned nothing, producing0.000%and a lag of minus the entire chain height. Unknown heights render-, with a known tip still shown alone. A real sync height of0— a daemon at the very start of initial sync — still prints its large negative lag. - The
eth_syncingcheck compared the response against an exact byte string, so any whitespace or field-order difference between providers fell through to the syncing branch. It now asks jq whether.resultisfalse. The syncing branch reports both heights but never set a progress figure, so the row printed an empty SYNC cell in the one state where progress is the interesting part. - The beacon endpoint was derived by rewriting the execution endpoint's port to 3500, which drops a remote URL's path and points at a port providers do not serve, so the row was permanently
Error. It is derived only for the in-cluster daemon, honors an explicitETH_BEACON_HOST, and otherwise renders as an absent measurement — the-the table already prints where a figure does not apply. A configured-but-unreachable beacon still readsError, so a missing beacon stays distinguishable from a broken one. - Chains disabled via
BIFROST_CHAINS_<C>_DISABLEDprinted0.000% 0 0, reading as broken rather than configured. Their rows are hidden, extending the gating XRP/TRON/SOL already had to every chain. - The AVAX reference endpoint (
api.avax.network) rejects requests from many hosting-provider ranges, so the row renderedErrorfor any node whose egress it blocks. Replaced with the PublicNode endpoint, which serves them. - The churn ETAs estimate how long a span of blocks takes by asking how long ago the block that far below the tip was produced. That mirrored block does not always exist: when churning is halted the next churn height is already past, so the mirror lands above the tip, and the ETA printed nonsense such as
20670d 18h— epoch seconds rendered as days. The same happens whenever the mirror predates what the node retains, which for a freshly synced node is everything. The overdue case is now detected and reported as such, and the estimate falls back to the network's own nominal rate,BlocksPerYear, when the mirrored block is unavailable. Measurement is still preferred where the block is there, since the chain runs slightly slower than nominal. - The
HALTCHURNINGmimir value is reported whether or not the churn is overdue: churning can be halted while the next churn height is still ahead, and a bare countdown to a churn that cannot happen reads as a schedule. - The guard that skips churn estimates while catching up compared with the wrong sign — it skipped when the node was ahead of the reference and never while actually behind — and the estimates now run only when both THOR heights are real numbers, since an unset height is
0in bash arithmetic and reads as a node at genesis.
Concurrency
Every probe is an independent round trip, so they are fetched concurrently into a scratch directory and the cached bodies parsed afterwards. The parsing is unchanged — each X_RESULT=$(curl ...) became X_RESULT=$(body x) — so the arithmetic, the guards and the rendering are the same code.
Beyond wall time, this makes the table a coherent snapshot rather than a sequence of observations. Every BEHIND cell is a difference between two heights, so any delay between the two requests behind it lands in that difference: read one after the other, a chain sitting exactly at tip could report a block of lag that was only the time elapsed between its own two requests, and the chains with the shortest blocks were the worst affected. Rows were not comparable with each other either, the first having been measured a whole table's worth of requests before the last, nor with the scanner status, which is a single instant. This applies to an in-cluster deployment as much as a remote one, since the reference endpoints are public in both cases.
Three probes needed more than a mechanical change: the THOR tip reference tries a peer, then each seed in order, then the network's public RPC, so that sequence runs as one background job; both Ethereum requests are issued upfront because which one answers depends on what eth_syncing returns; and the churn ETA queries a block computed from the heights, so it necessarily follows them.
Verification
Verified on mainnet against remote RPC endpoints, and in the thornode container for the fullnode path — with VALIDATOR=false the script makes no outbound chain requests at all, confirmed by tracing the requests actually issued. Every scanner state was exercised against recorded payloads: genesis sync, mid-sync, the unhealthy marker, the sentinel disambiguation, both fetch-failure modes, and the widest possible cells. shellcheck reports no findings at any severity, for every commit in the series individually. The churn estimates were checked in all three states: overdue, a churn near enough that the mirrored block is retained, and one far enough that it is not.
No behaviour changes for an in-cluster-daemon deployment beyond the table layout: scheme-less endpoints keep the credential prefix, the beacon row keeps its derivation, and reachable endpoints render as before.