feat(metrics): THORNode metric collection and anomaly scoring

Midgard scrapes THORNode's Prometheus metrics, plus a few facts Prometheus does not expose (invariants, per-asset solvency), into a TimescaleDB hypertable, and serves three endpoints that say how unusual each metric currently is against its own recent history.

Endpoints

GET /v2/metrics/scores Every series' newest completed hour, least common first.
GET /v2/metrics/anomalies The same list cut at ?below= (default 0.05), plus every series whose newest hour has no real samples. This is the one to page on.
GET /v2/metrics/data?series= Every scored hour for one series, oldest first. For looking closely.

All take ?window=24h|7d|14d (default 14d); /anomalies rejects 24h because its floor of 2/24 is above the default threshold. All return 404 when collection is disabled on the instance.

Every row carries:

field meaning
commonness Rank statistic in [2/n, 1]: the share of the window at least as extreme as this value, two-tailed, doubled. 1 is ordinary, 2/n is the single most extreme hour. Multiply by n/2 to read it as "hours in the window at least this extreme".
deltaCommonness The same, applied to the hour-over-hour change.
percentileRank Direction only: 0 is the low end of the window, 1 the high end.
modifiedZ Magnitude and sign: 0.6745 · (value − median) / MAD, in robust standard deviations. |z| ≥ 3.5 is the conventional cutoff. Median/MAD so one past spike cannot inflate the spread and hide the next; mean-absolute-deviation fallback when MAD is 0; 0 on a constant series.
samples Real readings in the hour. 0 means the value was carried forward, not observed.
bucketCount Hours in the window, n.
value Gauges are averaged over the hour; counters report the increase within it.

Every number is a JSON string, as everywhere else in this API.

How the score works

Samples roll up to hourly buckets (gauge = average, counter = max − min; a counter hour with one sample is dropped rather than reported as a fabricated zero). Every series gets a row per hour in the window, gaps filled with the last real reading. Each bucket is then ranked against every other bucket of its own series in one pass of window functions, partitioned by series, so a node exposing hundreds of series is scored in one query.

Commonness uses CUME_DIST, not PERCENT_RANK, so a run of identical readings scores as common rather than as an extreme. That is the property that makes it usable on network counters, which are flat most of the time.

Two things a rank cannot do, and the fields that cover them:

  • It cannot say how far out a value is: 1% past the window maximum and 100× it both score 2/n. modifiedZ carries that.
  • It cannot see a dead collector. Carried-forward hours are ties with the frozen value, and every tie pushes commonness up by at least 2/n, so on commonness alone a stale series looks healthier the longer it has been dead. samples exposes it, and /anomalies appends every series whose newest hour has samples = 0 regardless of score.

Configuration

key default
metrics.enabled auto auto probes scrape_url once at startup and enables collection if it answers. true/false force it.
metrics.scrape_url http://localhost:26660/metrics CometBFT's Prometheus endpoint.
metrics.scrape_interval 60s Must be ≤ 30m so counters get two samples an hour.
metrics.scrape_timeout 5s
metrics.invariant_interval 10m THORNode's invariant sweep iterates the whole KV store, so it runs less often than the scrape.
metrics.retention_days 30 0 keeps samples forever.

Storage

Samples live in their own midgard_metrics schema, outside the hashed DDL, so a chain re-sync does not destroy them. Unlike every other table, scraped samples cannot be rebuilt from blocks. Chunks are one day, compressed after one day (segmented by series), and dropped by a retention policy. No *.sql under internal/db/ is touched, so no DDL hash change and no re-sync.

Four series are derived from Midgard's own aggregates rather than scraped, and so have history back to genesis: swap_count, swap_volume_usd_e8, liquidity_fees_rune_e8, pool_balance_rune_e8.

Verification

  • go test -p 1 ./... green except TestNetworkAPY, which fails identically at the pre-branch commit (a float-formatting difference). Pre-existing.
  • golangci-lint run clean under CI's pinned golangci/golangci-lint:v1.64.5. A locally installed v2 cannot read this repo's v1 config and silently lints nothing.
  • make generated leaves the tree clean.
  • The end-to-end checks that do not need a live THORNode were run against the real api.Handler with a stub Prometheus endpoint: samples survive a full CoreDDL() re-apply; the auto-probe enables and disables correctly and the endpoints 404 when disabled; a flat gauge with one spike scores 1.0 and 2/n respectively; a rising counter reports its hourly increase, not its lifetime value; deleted hours are carried forward, not read as zero.

Cost, warm, on 300 series × 14 days with one sample an hour: /scores ≈ 900 ms (≈ 640 ms before the median/MAD aggregations were added), /data ≈ 5 ms. The whole-set query is memoised per (window, hour) behind one mutex, so one polling client costs one query an hour, not one per request.

Known limits and decisions for review

  1. The default threshold has not been calibrated. A rank flags a fixed share of its own history by construction: a freely varying series lands at or below 0.05 roughly 4.8% of the time at 14d whatever the network is doing. On 300 series that is on the order of 14 spurious entries an hour before discounting constant gauges. An empty /anomalies is therefore not proof of health. Measuring the real null rate needs a mainnet node; the default should be revisited once it is known.
  2. Stale-series reporting has two noise sources. A series that legitimately ends (a retired asset label, a removed invariant) is reported as stale every hour until it ages out of the window. And the aggregate-derived series have no row for a zero-activity hour, so on a quiet chain they read as stale. An "hours since last real sample" cap is the natural follow-up if either matters.
  3. Counter resets corrupt an hour. max − min over a THORNode restart reports roughly the counter's lifetime as one hour's increase, which looks like an enormous real spike. The plan's deferral says this would show as a large negative value; it cannot, max − min is non-negative. The detection plan needs correcting whether or not the rate()-style fix lands now.
  4. Cost at scale. At a 60 s scrape the rollup scans every sample in the window (≈ 6 M rows for 300 series over 14 d). A TimescaleDB continuous aggregate over (series, kind, hour) measured ~4× faster and is cheap to add, since this schema is outside the hashed DDL and set_integer_now_func is already installed.
  5. Should enabled: auto be the default for the first release, given 1 and 4, or should operators opt in?

Smaller: aggregate-derived series names are unprefixed and share a namespace with scraped ones; thornode_invariant_broken's "alert on value == 1" is unreachable through an hourly average; the INSERT is all-or-nothing above ~21,845 samples (bind-parameter cap); a failed retention policy warns once at startup then goes silent.

Before enabling on a real node

The series count drives every cost figure above and was not verifiable here:

curl -s localhost:26660/metrics | grep -c '^[a-z]'
curl -s $THORNODE/mimir | jq 'length'

🤖 Generated with Claude Code

https://claude.ai/code/session_01WoU3SuTW43Nykts7QenzPB

Edited by Son of Odin

Merge request reports

Loading
Loading