feat(metrics): THORNode metric collection and anomaly scoring
Midgard scrapes THORNode's Prometheus metrics, plus a few facts Prometheus does not expose (invariants, per-asset solvency), into a TimescaleDB hypertable, and serves three endpoints that say how unusual each metric currently is against its own recent history.
Endpoints
GET /v2/metrics/scores |
Every series' newest completed hour, least common first. |
GET /v2/metrics/anomalies |
The same list cut at ?below= (default 0.05), plus every series whose newest hour has no real samples. This is the one to page on. |
GET /v2/metrics/data?series= |
Every scored hour for one series, oldest first. For looking closely. |
All take ?window=24h|7d|14d (default 14d); /anomalies rejects 24h because its floor of 2/24 is above the default threshold. All return 404 when collection is disabled on the instance.
Every row carries:
| field | meaning |
|---|---|
commonness |
Rank statistic in [2/n, 1]: the share of the window at least as extreme as this value, two-tailed, doubled. 1 is ordinary, 2/n is the single most extreme hour. Multiply by n/2 to read it as "hours in the window at least this extreme". |
deltaCommonness |
The same, applied to the hour-over-hour change. |
percentileRank |
Direction only: 0 is the low end of the window, 1 the high end. |
modifiedZ |
Magnitude and sign: 0.6745 · (value − median) / MAD, in robust standard deviations. |z| ≥ 3.5 is the conventional cutoff. Median/MAD so one past spike cannot inflate the spread and hide the next; mean-absolute-deviation fallback when MAD is 0; 0 on a constant series. |
samples |
Real readings in the hour. 0 means the value was carried forward, not observed. |
bucketCount |
Hours in the window, n. |
value |
Gauges are averaged over the hour; counters report the increase within it. |
Every number is a JSON string, as everywhere else in this API.
How the score works
Samples roll up to hourly buckets (gauge = average, counter = max − min; a counter hour with one sample is dropped rather than reported as a fabricated zero). Every series gets a row per hour in the window, gaps filled with the last real reading. Each bucket is then ranked against every other bucket of its own series in one pass of window functions, partitioned by series, so a node exposing hundreds of series is scored in one query.
Commonness uses CUME_DIST, not PERCENT_RANK, so a run of identical readings scores as common rather than as an extreme. That is the property that makes it usable on network counters, which are flat most of the time.
Two things a rank cannot do, and the fields that cover them:
- It cannot say how far out a value is: 1% past the window maximum and 100× it both score 2/n.
modifiedZcarries that. - It cannot see a dead collector. Carried-forward hours are ties with the frozen value, and every tie pushes commonness up by at least 2/n, so on commonness alone a stale series looks healthier the longer it has been dead.
samplesexposes it, and/anomaliesappends every series whose newest hour hassamples = 0regardless of score.
Configuration
| key | default | |
|---|---|---|
metrics.enabled |
auto |
auto probes scrape_url once at startup and enables collection if it answers. true/false force it. |
metrics.scrape_url |
http://localhost:26660/metrics |
CometBFT's Prometheus endpoint. |
metrics.scrape_interval |
60s |
Must be ≤ 30m so counters get two samples an hour. |
metrics.scrape_timeout |
5s |
|
metrics.invariant_interval |
10m |
THORNode's invariant sweep iterates the whole KV store, so it runs less often than the scrape. |
metrics.retention_days |
30 |
0 keeps samples forever. |
Storage
Samples live in their own midgard_metrics schema, outside the hashed DDL, so a chain re-sync does not destroy them. Unlike every other table, scraped samples cannot be rebuilt from blocks. Chunks are one day, compressed after one day (segmented by series), and dropped by a retention policy. No *.sql under internal/db/ is touched, so no DDL hash change and no re-sync.
Four series are derived from Midgard's own aggregates rather than scraped, and so have history back to genesis: swap_count, swap_volume_usd_e8, liquidity_fees_rune_e8, pool_balance_rune_e8.
Verification
go test -p 1 ./...green exceptTestNetworkAPY, which fails identically at the pre-branch commit (a float-formatting difference). Pre-existing.golangci-lint runclean under CI's pinnedgolangci/golangci-lint:v1.64.5. A locally installed v2 cannot read this repo's v1 config and silently lints nothing.make generatedleaves the tree clean.- The end-to-end checks that do not need a live THORNode were run against the real
api.Handlerwith a stub Prometheus endpoint: samples survive a fullCoreDDL()re-apply; the auto-probe enables and disables correctly and the endpoints 404 when disabled; a flat gauge with one spike scores 1.0 and 2/n respectively; a rising counter reports its hourly increase, not its lifetime value; deleted hours are carried forward, not read as zero.
Cost, warm, on 300 series × 14 days with one sample an hour: /scores ≈ 900 ms (≈ 640 ms before the median/MAD aggregations were added), /data ≈ 5 ms. The whole-set query is memoised per (window, hour) behind one mutex, so one polling client costs one query an hour, not one per request.
Known limits and decisions for review
- The default threshold has not been calibrated. A rank flags a fixed share of its own history by construction: a freely varying series lands at or below 0.05 roughly 4.8% of the time at 14d whatever the network is doing. On 300 series that is on the order of 14 spurious entries an hour before discounting constant gauges. An empty
/anomaliesis therefore not proof of health. Measuring the real null rate needs a mainnet node; the default should be revisited once it is known. - Stale-series reporting has two noise sources. A series that legitimately ends (a retired asset label, a removed invariant) is reported as stale every hour until it ages out of the window. And the aggregate-derived series have no row for a zero-activity hour, so on a quiet chain they read as stale. An "hours since last real sample" cap is the natural follow-up if either matters.
- Counter resets corrupt an hour.
max − minover a THORNode restart reports roughly the counter's lifetime as one hour's increase, which looks like an enormous real spike. The plan's deferral says this would show as a large negative value; it cannot,max − minis non-negative. The detection plan needs correcting whether or not therate()-style fix lands now. - Cost at scale. At a 60 s scrape the rollup scans every sample in the window (≈ 6 M rows for 300 series over 14 d). A TimescaleDB continuous aggregate over
(series, kind, hour)measured ~4× faster and is cheap to add, since this schema is outside the hashed DDL andset_integer_now_funcis already installed. - Should
enabled: autobe the default for the first release, given 1 and 4, or should operators opt in?
Smaller: aggregate-derived series names are unprefixed and share a namespace with scraped ones; thornode_invariant_broken's "alert on value == 1" is unreachable through an hourly average; the INSERT is all-or-nothing above ~21,845 samples (bind-parameter cap); a failed retention policy warns once at startup then goes silent.
Before enabling on a real node
The series count drives every cost figure above and was not verifiable here:
curl -s localhost:26660/metrics | grep -c '^[a-z]'
curl -s $THORNODE/mimir | jq 'length'