node-status: fix false-healthy reporting, add bifrost scanner columns, issue probes concurrently

🔎 Visual review of 451da79c: explainer, review flags and prompts to ask an AI about the change

Summary

make status renders several failure modes as success — most seriously, a chain whose endpoint answered nothing at all shows 100.000%. The endpoint fallbacks added for BIFROST_CHAINS_*_RPC_HOST make this easy to hit, because those endpoints are full URLs rather than the bare host:port of an in-cluster daemon. The table also says nothing about whether bifrost itself is keeping up with the daemons it observes, and the probes were issued one at a time, so no two rows described the same instant.

Four commits, each reviewable on its own:

  1. Accuracy — every path that rendered a failure as health, plus the table restructure
  2. Concurrency — issue the independent probes together
  3. JSON-RPC error bodies — treat an error response as a missing height, not the string null
  4. TRON probe gating — a fullnode no longer queries TRON endpoints for a row it does not print

The table

SYNC and BEHIND are now a pair reported per component, ordered so each column is measured against the one to its right: bifrost's block scanner trails the daemon, the daemon trails the TIP reference. Negative BEHIND means behind in both.

                   BIFROST                   DAEMON
CHAIN              SYNC       BEHIND         SYNC       BEHIND         TIP
THOR               -          -              100.000%   0              27,326,852
BTC                100.000%   0              100.000%   0              961,390
ETH                100.000%   -1             100.000%   0              25,701,482
ETH (beacon slot)  -          -              -          -              -
GAIA               100.000%   -1             100.000%   0              32,387,127
BSC                100.000%   -13            100.000%   0              114,502,014
...

The BIFROST pair comes from /status/scanner and reports what the daemon columns structurally cannot: a daemon sitting exactly at tip whose scanner is stalled or catching up previously read 100.000% 0, and that lag is what accrues observation slash points. The cells distinguish the states bifrost reports — a lag in blocks, Error for an unreadable height, missing for an enabled chain bifrost holds no client for (its endpoint was unreachable when bifrost started, which becomes invisible once the endpoint recovers), and an asterisk plus footnote when the scanner reports itself unhealthy. THOR has no scanner height to report and a fullnode has no bifrost, so both read -.

The endpoint is requested once and cached; every row parses that one response.

Two details of reading that endpoint are worth noting. Bifrost reports -1 as its could-not-read sentinel, but scanner_height_diff also takes -1 in ordinary operation: the handler fetches the chain height live while the scanner height comes from local storage, so a scanner one block ahead of the tip GetHeight() happened to return — ordinary skew when the endpoint is load-balanced — reports a diff of exactly -1. The sentinel is therefore tested on the heights, and BEHIND is computed from those two heights in the daemon column's sign convention rather than from the reported diff. All three fields are selected in one pass with @tsv, since a separate jq invocation per field is a process apiece for every row.

Accuracy fixes

  • calc_progress returned 100.000% whenever both heights were empty, because "" = "" compares equal, so an unreachable endpoint rendered as fully synced. A zero reference height rendered inf%. Heights are now accepted only as plain non-negative integers.
  • A JSON-RPC error body — a rate limit, a revoked key — is served with HTTP 200, so curl --fail passes it through, and jq -r prints the literal string null for the absent field. That compared equal exactly as the empty strings did. All heights are now extracted through a helper that yields nothing for a missing field, which also fixes SOL, whose .result // 0 turned a dead endpoint into height 0, and the EVM chains, which fed null to printf %.0f and got a confident 0 plus an error on stderr.
  • The UTXO probes prefixed http://thorchain:<redacted>@ onto the endpoint for the in-cluster daemons' credentials. Applied to a URL that already carries a scheme the request is malformed and returns nothing, which the above then rendered as 100%. The prefix now applies only to scheme-less endpoints.
  • Several chains substituted a sync height of 0 when their endpoint returned nothing, producing 0.000% and a lag of minus the entire chain height. Unknown heights render -, with a known tip still shown alone. A real sync height of 0 — a daemon at the very start of initial sync — still prints its large negative lag.
  • The eth_syncing check compared the response against an exact byte string, so any whitespace or field-order difference between providers fell through to the syncing branch. It now asks jq whether .result is false. The syncing branch reports both heights but never set a progress figure, so the row printed an empty SYNC cell in the one state where progress is the interesting part.
  • The beacon endpoint was derived by rewriting the execution endpoint's port to 3500, which drops a remote URL's path and points at a port providers do not serve, so the row was permanently Error. It is derived only for the in-cluster daemon, honors an explicit ETH_BEACON_HOST, and otherwise renders as an absent measurement — the - the table already prints where a figure does not apply. A configured-but-unreachable beacon still reads Error, so a missing beacon stays distinguishable from a broken one.
  • Chains disabled via BIFROST_CHAINS_<C>_DISABLED printed 0.000% 0 0, reading as broken rather than configured. Their rows are hidden, extending the gating XRP/TRON/SOL already had to every chain.
  • The AVAX reference endpoint (api.avax.network) rejects requests from many hosting-provider ranges, so the row rendered Error for any node whose egress it blocks. Replaced with the PublicNode endpoint, which serves them.
  • The churn ETAs estimate how long a span of blocks takes by asking how long ago the block that far below the tip was produced. That mirrored block does not always exist: when churning is halted the next churn height is already past, so the mirror lands above the tip, and the ETA printed nonsense such as 20670d 18h — epoch seconds rendered as days. The same happens whenever the mirror predates what the node retains, which for a freshly synced node is everything. The overdue case is now detected and reported as such, and the estimate falls back to the network's own nominal rate, BlocksPerYear, when the mirrored block is unavailable. Measurement is still preferred where the block is there, since the chain runs slightly slower than nominal.
  • The HALTCHURNING mimir value is reported whether or not the churn is overdue: churning can be halted while the next churn height is still ahead, and a bare countdown to a churn that cannot happen reads as a schedule.
  • The guard that skips churn estimates while catching up compared with the wrong sign — it skipped when the node was ahead of the reference and never while actually behind — and the estimates now run only when both THOR heights are real numbers, since an unset height is 0 in bash arithmetic and reads as a node at genesis.

Concurrency

Every probe is an independent round trip, so they are fetched concurrently into a scratch directory and the cached bodies parsed afterwards. The parsing is unchanged — each X_RESULT=$(curl ...) became X_RESULT=$(body x) — so the arithmetic, the guards and the rendering are the same code.

Beyond wall time, this makes the table a coherent snapshot rather than a sequence of observations. Every BEHIND cell is a difference between two heights, so any delay between the two requests behind it lands in that difference: read one after the other, a chain sitting exactly at tip could report a block of lag that was only the time elapsed between its own two requests, and the chains with the shortest blocks were the worst affected. Rows were not comparable with each other either, the first having been measured a whole table's worth of requests before the last, nor with the scanner status, which is a single instant. This applies to an in-cluster deployment as much as a remote one, since the reference endpoints are public in both cases.

Three probes needed more than a mechanical change: the THOR tip reference tries a peer, then each seed in order, then the network's public RPC, so that sequence runs as one background job; both Ethereum requests are issued upfront because which one answers depends on what eth_syncing returns; and the churn ETA queries a block computed from the heights, so it necessarily follows them.

Verification

Verified on mainnet against remote RPC endpoints, and in the thornode container for the fullnode path — with VALIDATOR=false the script makes no outbound chain requests at all, confirmed by tracing the requests actually issued. Every scanner state was exercised against recorded payloads: genesis sync, mid-sync, the unhealthy marker, the sentinel disambiguation, both fetch-failure modes, and the widest possible cells. shellcheck reports no findings at any severity, for every commit in the series individually. The churn estimates were checked in all three states: overdue, a churn near enough that the mirrored block is retained, and one far enough that it is not.

No behaviour changes for an in-cluster-daemon deployment beyond the table layout: scheme-less endpoints keep the credential prefix, the beacon row keeps its derivation, and reachable endpoints render as before.

Supersedes !5003 (closed) (moved from fork)

Edited by Huginn Ai

Merge request reports

Loading
Loading