feat(skills): add tc-metrics for Midgard metric anomaly triage
🔎 Visual review of8f53933f: explainer, review flags and prompts to ask an AI about the change
Adds a tc-metrics Claude Code skill on top of the anomaly-scoring endpoints from midgard!733 (merged).
.claude/skills/tc-metrics/metrics.py ranks every series (736 today) by how unusual its newest hour is. Stale rows come first, then commonness ascending, then |modifiedZ| descending. --grep restricts the ranking to matching series, and --series prints one series' full hourly history. The script also handles the things that make the raw endpoint misleading in practice:
| Trap | Handling |
|---|---|
/anomalies returns only samples=0 series while history is short: most series' 2/n floor sits above the 0.05 default until n ≥ 40 (and 24h is rejected outright below 0.083) |
Detected from the actual bucket counts and reported in the header; ranks /scores directly instead |
bucketCount is per-series, so commonness is not comparable across rows |
Header prints the bucket range and the most common n; exposed as an n column; n < 8 flagged |
MAD collapses to ~0 on near-constant series, giving |z| up to 1e14 |
Flagged, never dismissed: "was flat, moved this hour" when deltaCommonness is also low, otherwise "level shifted within window; --series for when" |
samples = 0 means that series' source failed, and ties push commonness up, so a frozen series scores as increasingly healthy |
Flagged STALE, counted in the header and sorted above the ranking so --top cannot hide it |
thornode_invariant_broken / thornode_solvency_excess_e8 mean something regardless of rank |
Shown unconditionally in a "read by value, not by rank" block |
| A 404 means collection is disabled on that Midgard, not that the network is fine | Reported as such; no fallback host |
SKILL.md splits series into three groups:
- Node-local:
go_*,process_*,promhttp_*, and the p2p/mempool/blocksync/consensus_step_durationseries describe only the Midgard host's thornode. - Seen by one node: the other
tendermint_/cometbft_consensus series need corroboration before they count as network findings. - Network-wide:
thornode_*(all four families come from THORNode REST, not the Prometheus scrape, so they go stale independently of it),swap_*,pool_*andliquidity_*(Midgard DB).
It also covers:
- the ~2h staleness, so findings are framed as "what was unusual last hour" and not "what is happening now";
- the hour-average
valuetrap: confirm levels against the THORNode API before quoting them; - the row shapes that need a second look: per-proposer delays, mimirs, monotone trends like
tendermint_consensus_height.
health gets an optional step 1b discovery sweep. It runs metrics.py with a grep that drops node-local and per-proposer series, explains how to read the output, uses survivors to narrow the Loki windows, and adds a "Metric Anomalies" report table. Any network-wide survivor that stays unexplained makes the report at least tc-debug gets a one-line pointer here for the "no specific target, something just feels off" case.
Also bumps the Go toolchain from 1.26.9 to 1.26.10 for GO-2026-6809: crypto/x509 now distrusts compromised intermediate certificates, and govulncheck reports this as reachable through the monero sidecar's net/http server. The bump covers go.mod, .go-version, CI and Docker images, trunk, scripts and docs, plus CI_TOOLS_VERSION so the tools image gets rebuilt.
Verified end to end against https://gateway-lon.liquify.com/chain/thorchain_midgard/v2/metrics.