ci(test-integrity): summarise regression golden changes per field

🔎 Visual review: building it for 9ec396c6; the link appears here.

What and why

test-integrity makes a human add goldens-reviewed before golden changes merge, but the reviewer still gets thousands of lines of JSON diff. Wrong behavior has been approved inside such diffs:

New scripts/golden-diff-summary.py uses only the stdlib. It takes two revisions, or in CI the same MR diff base and head as test-integrity. It does this:

  • Parses every changed file under test/regression/mnt/exports/ and blocks/ on both sides and diffs them structurally:
    • blocks are paired by height;
    • array elements are paired by their id fields, then by those fields ignoring case, then (for block txs) by message types;
    • identical elements are skipped.
  • Groups the changes by normalised path, for example blocks[].end_block_events[type=swap].streaming_swap_count. For each group it reports the changed values, the files and suites, the top old→new transitions, and the keys or elements added and removed.
  • Flags and ranks the groups:
    • COLLAPSE: many old values became one value (all of them, or at least half from three or more sources);
    • ZEROED, SIGN and CASE;
    • UNIFORM, UP and DOWN;
    • GONE: elements removed and none added;
    • UNTOUCHED: suites whose YAML the MR does not touch.
  • Ranks last the fields the harness pins (app_hash, last_results_hash and gas_used, since 0fe1fcc8) and opaque hashes.
  • Writes a markdown summary of 150 lines or fewer, plus a full-detail file with every group and example locations (file@h<height>).

CI: the test-integrity job runs the summary after the guard and prints it. It also saves golden-diff-summary.md and golden-diff-detail.md as the "golden diff summary" artifact (expose_as, so it appears in the MR widget). The job's result is still the guard's exit code; the summary never fails it. When goldens changed, the guard's failure message now points at the summary. REVIEW.md gate 4, its checklist and README.md now ask reviewers to read the summary and explain every flagged row before they add goldens-reviewed.

Testing

  • python3 -m unittest discover -s scripts/tests -p 'test_*_test.py': 36 tests pass, 17 of them new. The new tests use synthetic JSON pairs and cover:
    • a collapse that ranks above benign amount drift, a majority collapse, and a uniform shift that is not a collapse;
    • an inserted event that does not shift pairs;
    • a lowercase TxID that shows as CASE, not OPAQUE;
    • a changed mimir key, which is a removal plus an addition;
    • sign flips, zeroed values, and removed keys;
    • pinned fields ranking last, untouched suites, block heights in examples, the line cap, and pipe escaping;
    • end to end against a temporary git repo: new-suite, added, deleted and unparseable files, no golden changes, and a bad revision.
  • glab ci lint: valid. TMPDIR=$(mktemp -d) make lint-ci: clean (exit 0).
  • Real runs on history:
    • c0d01c49 (!4140 (merged)): the two collapses are rows 1 and 2, and both were real bugs. Full output is below.
    • Fix commits: 0d15e009 (!4728 (merged)) shows the TxID case change at the top (CASE×1 on the outbound memo and the swap id). 0cfecfd6 (!4394 (merged)) shows streaming_swap_count going back up (1→2, 1→3, …), with no collapse flag.
    • Benign bulk updates stay at one to three rows and raise no value flags:
      • 4282a76f (SOL nonces, 285 files): +[] ×184 and +null ×425;
      • ceea8852 (171 files): vaults[type=AsgardVault].deficit −null ×173;
      • d517826a (remove authz, 181 files): export.app_state.authz −{…} ×181.
    • 0fe1fcc8, which made the harness pin static values, shows its three groups as PINNED, ranked last.
python3 scripts/golden-diff-summary.py c0d01c49aa^ c0d01c49aa

Regression golden changes, by field

7198a5975fb3..c0d01c49aa24: 95 golden files diffed in 66 suites (42 whose suite YAML this diff does not touch). 7124 values changed, 1694 keys or elements added, 1652 removed, in 200 field groups. Added with a new suite (not diffed): 2 file(s): test/regression/mnt/blocks/adv-swap-queue/preferred-asset-swaps.json, test/regression/mnt/blocks/adv-swap-queue/streaming-limit-swaps.json. Deleted: 6 file(s): test/regression/mnt/blocks/lending/affiliates.json, test/regression/mnt/blocks/lending/uncommon.json, test/regression/mnt/blocks/pools/check-savers-utilisation.json, test/regression/mnt/blocks/ragnarok/ragnarok.json and 2 more.

Before adding goldens-reviewed, confirm every flagged row is intended. COLLAPSE, ZEROED, SIGN and CASE are how wrong behavior has been pinned before (!4140 (merged) → !4394 (merged), !4728 (merged), !4730 (merged)).

# Field Changed +/− Files/suites Top transitions (or values added/removed) Flags
1 blocks[].end_block_events[type=swap].streaming_swap_quantity 96 6/6 9→1 ×36, 10→1 ×20, 9→11 ×18, +2 more COLLAPSE→1 81% UNTOUCHED 2/6
2 blocks[].end_block_events[type=swap].streaming_swap_count 206 11/11 2→1 ×39, 3→1 ×18, 4→1 ×17, +24 more COLLAPSE→1 68% DOWN UNTOUCHED 2/11
3 blocks[].end_block_events[type=scheduled_outbound].coin_decimals 14 5/5 8→0 ×14 UNIFORM ZEROED DOWN UNTOUCHED 4/5
4 blocks[].end_block_events[type=rewards].ETH.ETH 55 +21 −19 15/15 347768→347751 ×2, 347755→347768 ×2, 347759→347772 ×2, +48 more UNTOUCHED 2/15
5 blocks[].end_block_events[type=coin_received].amount 806 19/19 199800rune→199771rune ×5, 347755rune→347768rune ×4, 66591rune→66588rune ×4, +677 more UNTOUCHED 3/19
6 blocks[].end_block_events[type=coin_spent].amount 757 19/19 4609267eth-eth→4609268eth-eth ×6, 199800rune→199771rune ×5, 347755rune→347768rune ×4, +674 more UNTOUCHED 3/19
7 blocks[].end_block_events[type=transfer].amount 740 19/19 199800rune→199771rune ×5, 347755rune→347768rune ×4, 66591rune→66588rune ×4, +673 more UNTOUCHED 3/19
8 blocks[].end_block_events[type=rewards].BTC.BTC 124 +4 −4 16/16 347719→347751 ×2, 199800→199771 ×1, 347743→347719 ×1, +120 more UNTOUCHED 1/16
9 blocks[].end_block_events[type=coin_received] +171 −220 17/17 −{…} ×220, +{…} ×171 UNTOUCHED 3/17
10 blocks[].end_block_events[type=coin_spent] +163 −210 17/17 −{…} ×210, +{…} ×163 UNTOUCHED 3/17
11 blocks[].end_block_events[type=message] +128 −170 17/17 −{…} ×170, +{…} ×128 UNTOUCHED 3/17
12 blocks[].end_block_events[type=transfer] +128 −170 17/17 −{…} ×170, +{…} ×128 UNTOUCHED 3/17
13 blocks[].end_block_events[type=rewards].bond_reward 141 15/15 66588→66591 ×8, 66591→66588 ×4, 66591→66589 ×4, +125 more UNTOUCHED 2/15
14 blocks[].end_block_events[type=swap].emit_asset 131 8/8 497740806 THO…→497740914 THO… ×2, 493013242 THO…→493013347 THO… ×2, 483758274 THO…→483758271 THO… ×2, +120 more UNTOUCHED 2/8
15 export.app_state.thorchain.observed_tx_in_voters[].actions[].coin.decimals −1 1/1 −8 ×1 GONE×1 UNTOUCHED 1/1
16 export.app_state.thorchain.tx_outs[].tx_array[].coin.decimals −1 1/1 −8 ×1 GONE×1 UNTOUCHED 1/1
17 blocks[].end_block_events[type=swap].liquidity_fee_in_rune 107 9/9 2477453→2477454 ×2, 2320258→2320257 ×2, 2438650→2475215 ×1, +102 more UNTOUCHED 2/9
18 blocks[].end_block_events[type=swap].coin 83 8/8 4609267 ETH-E…→4609268 ETH-E… ×3, 5185425 ETHE…→5185426 ETHE… ×3, 4609267 ETHE…→4609268 ETHE… ×3, +65 more UNTOUCHED 2/8
19 export.app_state.bank.balances[].coins[].amount 76 21/21 35000269880252→35010021500753 ×1, 2499945708680→2492847448814 ×1, 2499794405724→2449998000000 ×1, +73 more UNTOUCHED 2/21
20 blocks[].end_block_events[type=swap].liquidity_fee 76 8/8 2477453→2477454 ×2, 2320258→2320257 ×2, 2438650→2475215 ×1, +71 more UNTOUCHED 2/8
21 export.app_state.thorchain.pools[].balance_rune 69 18/18 100001198660→100001398431 ×5, 100001398426→100001598197 ×4, 100000998860→100001198631 ×3, +49 more UNTOUCHED 2/18
22 blocks[].end_block_events[type=rewards].ETH.USDX-0X9999999999999999999999999999999999999999 14 +14 −13 5/5 199800→199771 ×1, 199778→199772 ×1, 200069→200055 ×1, +11 more
23 blocks[].end_block_events[type=mint_burn] +86 −95 13/13 −{…} ×95, +{…} ×86 UNTOUCHED 1/13
24 blocks[].txs[] +119 −123 12/12 −{…} ×123, +{…} ×119
25 blocks[].end_block_events[type=swap] +88 −92 12/12 −{…} ×92, +{…} ×88 UNTOUCHED 1/12
26 blocks[].begin_block_events[type=pool_balance_change].rune_amt 149 7/7 199772→100000799088 ×6, 347751→100001043253 ×3, 199772→100000998860 ×3, +127 more
27 blocks[].begin_block_events[type=pool_balance_change].asset_amt 126 7/7 0→1 ×10, 1→0 ×7, 0→99999999999 ×5, +95 more
28 blocks[].end_block_events[type=fee] +34 −49 14/14 −{…} ×49, +{…} ×34 UNTOUCHED 2/14
29 export.app_state.thorchain.outbound_fee_withheld_rune[].amount 17 12/12 201064→82258436 ×1, 15227786→75034897 ×1, 140169310→552465481 ×1, +14 more UP UNTOUCHED 1/12
30 export.app_state.thorchain.network.total_bond_units 15 15/15 9→11 ×1, 7→21 ×1, 6→58 ×1, +12 more UP

170 more field groups in the full detail.

Flags: COLLAPSE = many old values became one; ZEROED = all became 0 or empty; SIGN = sign flip; CASE = letter case only; UNIFORM = one old→new pair; UP/DOWN = every number moved one way; GONE = elements removed, none added; UNTOUCHED = suites whose YAML the diff does not touch; PINNED = static in the harness unless UNFILTER_EXPORT=1; OPAQUE = hashes or signatures.

Full detail (every field, examples by file and block): c0d-detail.md.

Benign: python3 scripts/golden-diff-summary.py 4282a76fec^ 4282a76fec (SOL durable-nonce outbounds, 285 golden files)

Regression golden changes, by field

c3d15b5dd54c..4282a76fecfa: 285 golden files diffed in 184 suites (184 whose suite YAML this diff does not touch). 0 values changed, 609 keys or elements added, 0 removed, in 3 field groups.

Before adding goldens-reviewed, confirm every flagged row is intended. COLLAPSE, ZEROED, SIGN and CASE are how wrong behavior has been pinned before (!4140 (merged) → !4394 (merged), !4728 (merged), !4730 (merged)).

# Field Changed +/− Files/suites Top transitions (or values added/removed) Flags
1 export.app_state.thorchain.sol_vault_nonces +184 184/184 +[] ×184
2 blocks[].txs[].tx.body.messages[@type=/types.MsgObservedTxIn].txs[].sol_nonce +267 87/87 +null ×267
3 blocks[].txs[].tx.body.messages[@type=/types.MsgObservedTxOut].txs[].sol_nonce +158 53/53 +null ×158

Flags: COLLAPSE = many old values became one; ZEROED = all became 0 or empty; SIGN = sign flip; CASE = letter case only; UNIFORM = one old→new pair; UP/DOWN = every number moved one way; GONE = elements removed, none added; UNTOUCHED = suites whose YAML the diff does not touch; PINNED = static in the harness unless UNFILTER_EXPORT=1; OPAQUE = hashes or signatures.

What it does not cover

  • It does not post an MR comment. CI has no existing token mechanism for posting MR notes (the visual-review job only records an environment; huginn posts from outside CI), so it prints to the log and saves an artifact.
  • It does not diff goldens that are added or deleted; it lists them. That includes goldens added with a new suite.
  • Pairing is heuristic. When an array gains or loses events, some elements show as added or removed rather than changed, and generic bank events (coin_spent, transfer and so on) can appear as balanced +N −M rows.
  • It flags patterns, not intent. A deliberate collapse, such as a constant set to 1, is still flagged and needs a sentence in the MR.

Follow-ups

  • Post or update a single MR comment with the summary once a CI token for MR notes exists. huginn could post it instead.
  • Optionally run the summary in the regression job on EXPORT=1 diffs, so authors see it before they push.
  • 14 refund scheduled_outbound.coin_decimals values went 8→0 in !4140 (merged) (row 3 above). Most of them are still 0 on develop. Decimals 0 probably means "chain default", but someone should confirm.

🤖 Generated with Claude Code

Edited by Huginn Ai

Merge request reports

Loading
Loading