ci(test-integrity): summarise regression golden changes per field
🔎 Visual review: building it for9ec396c6; the link appears here.
What and why
test-integrity makes a human add goldens-reviewed before golden changes merge, but the reviewer still gets thousands of lines of JSON diff. Wrong behavior has been approved inside such diffs:
- !4140 (merged) (c0d01c49, 201 files) regenerated block goldens in which
streaming_swap_countandstreaming_swap_quantitycollapsed to"1". Count was fixed by !4394 (merged); quantity was restored later by 79728c2b. - !4728 (merged): the over-solvency golden had pinned a lowercase TxID.
- !4730 (merged): the gas and slash goldens had pinned under-slashed bonds.
- !4655 (merged) and !4734 (merged): regression goldens had pinned the wrong swap-queue order.
New scripts/golden-diff-summary.py uses only the stdlib. It takes two revisions, or in CI the same MR diff base and head as test-integrity. It does this:
- Parses every changed file under
test/regression/mnt/exports/andblocks/on both sides and diffs them structurally:- blocks are paired by height;
- array elements are paired by their id fields, then by those fields ignoring case, then (for block txs) by message types;
- identical elements are skipped.
- Groups the changes by normalised path, for example
blocks[].end_block_events[type=swap].streaming_swap_count. For each group it reports the changed values, the files and suites, the top old→new transitions, and the keys or elements added and removed. - Flags and ranks the groups:
COLLAPSE: many old values became one value (all of them, or at least half from three or more sources);ZEROED,SIGNandCASE;UNIFORM,UPandDOWN;GONE: elements removed and none added;UNTOUCHED: suites whose YAML the MR does not touch.
- Ranks last the fields the harness pins (
app_hash,last_results_hashandgas_used, since 0fe1fcc8) and opaque hashes. - Writes a markdown summary of 150 lines or fewer, plus a full-detail file with every group and example locations (
file@h<height>).
CI: the test-integrity job runs the summary after the guard and prints it. It also saves golden-diff-summary.md and golden-diff-detail.md as the "golden diff summary" artifact (expose_as, so it appears in the MR widget). The job's result is still the guard's exit code; the summary never fails it. When goldens changed, the guard's failure message now points at the summary.
REVIEW.md gate 4, its checklist and README.md now ask reviewers to read the summary and explain every flagged row before they add goldens-reviewed.
Testing
python3 -m unittest discover -s scripts/tests -p 'test_*_test.py': 36 tests pass, 17 of them new. The new tests use synthetic JSON pairs and cover:- a collapse that ranks above benign amount drift, a majority collapse, and a uniform shift that is not a collapse;
- an inserted event that does not shift pairs;
- a lowercase TxID that shows as
CASE, notOPAQUE; - a changed mimir key, which is a removal plus an addition;
- sign flips, zeroed values, and removed keys;
- pinned fields ranking last, untouched suites, block heights in examples, the line cap, and pipe escaping;
- end to end against a temporary git repo: new-suite, added, deleted and unparseable files, no golden changes, and a bad revision.
glab ci lint: valid.TMPDIR=$(mktemp -d) make lint-ci: clean (exit 0).- Real runs on history:
- c0d01c49 (!4140 (merged)): the two collapses are rows 1 and 2, and both were real bugs. Full output is below.
- Fix commits: 0d15e009 (!4728 (merged)) shows the TxID case change at the top (
CASE×1on the outbound memo and the swap id). 0cfecfd6 (!4394 (merged)) showsstreaming_swap_countgoing back up (1→2,1→3, …), with no collapse flag. - Benign bulk updates stay at one to three rows and raise no value flags:
- 0fe1fcc8, which made the harness pin static values, shows its three groups as
PINNED, ranked last.
python3 scripts/golden-diff-summary.py c0d01c49aa^ c0d01c49aa
Regression golden changes, by field
7198a5975fb3..c0d01c49aa24: 95 golden files diffed in 66 suites (42 whose suite YAML this diff does not touch). 7124 values changed, 1694 keys or elements added, 1652 removed, in 200 field groups.
Added with a new suite (not diffed): 2 file(s): test/regression/mnt/blocks/adv-swap-queue/preferred-asset-swaps.json, test/regression/mnt/blocks/adv-swap-queue/streaming-limit-swaps.json.
Deleted: 6 file(s): test/regression/mnt/blocks/lending/affiliates.json, test/regression/mnt/blocks/lending/uncommon.json, test/regression/mnt/blocks/pools/check-savers-utilisation.json, test/regression/mnt/blocks/ragnarok/ragnarok.json and 2 more.
Before adding goldens-reviewed, confirm every flagged row is intended. COLLAPSE, ZEROED, SIGN and CASE are how wrong behavior has been pinned before (!4140 (merged) → !4394 (merged), !4728 (merged), !4730 (merged)).
| # | Field | Changed | +/− | Files/suites | Top transitions (or values added/removed) | Flags |
|---|---|---|---|---|---|---|
| 1 | blocks[].end_block_events[type=swap].streaming_swap_quantity |
96 | 6/6 | 9→1 ×36, 10→1 ×20, 9→11 ×18, +2 more | COLLAPSE→1 81% UNTOUCHED 2/6 | |
| 2 | blocks[].end_block_events[type=swap].streaming_swap_count |
206 | 11/11 | 2→1 ×39, 3→1 ×18, 4→1 ×17, +24 more | COLLAPSE→1 68% DOWN UNTOUCHED 2/11 | |
| 3 | blocks[].end_block_events[type=scheduled_outbound].coin_decimals |
14 | 5/5 | 8→0 ×14 | UNIFORM ZEROED DOWN UNTOUCHED 4/5 | |
| 4 | blocks[].end_block_events[type=rewards].ETH.ETH |
55 | +21 −19 | 15/15 | 347768→347751 ×2, 347755→347768 ×2, 347759→347772 ×2, +48 more | UNTOUCHED 2/15 |
| 5 | blocks[].end_block_events[type=coin_received].amount |
806 | 19/19 | 199800rune→199771rune ×5, 347755rune→347768rune ×4, 66591rune→66588rune ×4, +677 more | UNTOUCHED 3/19 | |
| 6 | blocks[].end_block_events[type=coin_spent].amount |
757 | 19/19 | 4609267eth-eth→4609268eth-eth ×6, 199800rune→199771rune ×5, 347755rune→347768rune ×4, +674 more | UNTOUCHED 3/19 | |
| 7 | blocks[].end_block_events[type=transfer].amount |
740 | 19/19 | 199800rune→199771rune ×5, 347755rune→347768rune ×4, 66591rune→66588rune ×4, +673 more | UNTOUCHED 3/19 | |
| 8 | blocks[].end_block_events[type=rewards].BTC.BTC |
124 | +4 −4 | 16/16 | 347719→347751 ×2, 199800→199771 ×1, 347743→347719 ×1, +120 more | UNTOUCHED 1/16 |
| 9 | blocks[].end_block_events[type=coin_received] |
+171 −220 | 17/17 | −{…} ×220, +{…} ×171 | UNTOUCHED 3/17 | |
| 10 | blocks[].end_block_events[type=coin_spent] |
+163 −210 | 17/17 | −{…} ×210, +{…} ×163 | UNTOUCHED 3/17 | |
| 11 | blocks[].end_block_events[type=message] |
+128 −170 | 17/17 | −{…} ×170, +{…} ×128 | UNTOUCHED 3/17 | |
| 12 | blocks[].end_block_events[type=transfer] |
+128 −170 | 17/17 | −{…} ×170, +{…} ×128 | UNTOUCHED 3/17 | |
| 13 | blocks[].end_block_events[type=rewards].bond_reward |
141 | 15/15 | 66588→66591 ×8, 66591→66588 ×4, 66591→66589 ×4, +125 more | UNTOUCHED 2/15 | |
| 14 | blocks[].end_block_events[type=swap].emit_asset |
131 | 8/8 | 497740806 THO…→497740914 THO… ×2, 493013242 THO…→493013347 THO… ×2, 483758274 THO…→483758271 THO… ×2, +120 more | UNTOUCHED 2/8 | |
| 15 | export.app_state.thorchain.observed_tx_in_voters[].actions[].coin.decimals |
−1 | 1/1 | −8 ×1 | GONE×1 UNTOUCHED 1/1 | |
| 16 | export.app_state.thorchain.tx_outs[].tx_array[].coin.decimals |
−1 | 1/1 | −8 ×1 | GONE×1 UNTOUCHED 1/1 | |
| 17 | blocks[].end_block_events[type=swap].liquidity_fee_in_rune |
107 | 9/9 | 2477453→2477454 ×2, 2320258→2320257 ×2, 2438650→2475215 ×1, +102 more | UNTOUCHED 2/9 | |
| 18 | blocks[].end_block_events[type=swap].coin |
83 | 8/8 | 4609267 ETH-E…→4609268 ETH-E… ×3, 5185425 ETH |
UNTOUCHED 2/8 | |
| 19 | export.app_state.bank.balances[].coins[].amount |
76 | 21/21 | 35000269880252→35010021500753 ×1, 2499945708680→2492847448814 ×1, 2499794405724→2449998000000 ×1, +73 more | UNTOUCHED 2/21 | |
| 20 | blocks[].end_block_events[type=swap].liquidity_fee |
76 | 8/8 | 2477453→2477454 ×2, 2320258→2320257 ×2, 2438650→2475215 ×1, +71 more | UNTOUCHED 2/8 | |
| 21 | export.app_state.thorchain.pools[].balance_rune |
69 | 18/18 | 100001198660→100001398431 ×5, 100001398426→100001598197 ×4, 100000998860→100001198631 ×3, +49 more | UNTOUCHED 2/18 | |
| 22 | blocks[].end_block_events[type=rewards].ETH.USDX-0X9999999999999999999999999999999999999999 |
14 | +14 −13 | 5/5 | 199800→199771 ×1, 199778→199772 ×1, 200069→200055 ×1, +11 more | |
| 23 | blocks[].end_block_events[type=mint_burn] |
+86 −95 | 13/13 | −{…} ×95, +{…} ×86 | UNTOUCHED 1/13 | |
| 24 | blocks[].txs[] |
+119 −123 | 12/12 | −{…} ×123, +{…} ×119 | ||
| 25 | blocks[].end_block_events[type=swap] |
+88 −92 | 12/12 | −{…} ×92, +{…} ×88 | UNTOUCHED 1/12 | |
| 26 | blocks[].begin_block_events[type=pool_balance_change].rune_amt |
149 | 7/7 | 199772→100000799088 ×6, 347751→100001043253 ×3, 199772→100000998860 ×3, +127 more | ||
| 27 | blocks[].begin_block_events[type=pool_balance_change].asset_amt |
126 | 7/7 | 0→1 ×10, 1→0 ×7, 0→99999999999 ×5, +95 more | ||
| 28 | blocks[].end_block_events[type=fee] |
+34 −49 | 14/14 | −{…} ×49, +{…} ×34 | UNTOUCHED 2/14 | |
| 29 | export.app_state.thorchain.outbound_fee_withheld_rune[].amount |
17 | 12/12 | 201064→82258436 ×1, 15227786→75034897 ×1, 140169310→552465481 ×1, +14 more | UP UNTOUCHED 1/12 | |
| 30 | export.app_state.thorchain.network.total_bond_units |
15 | 15/15 | 9→11 ×1, 7→21 ×1, 6→58 ×1, +12 more | UP |
170 more field groups in the full detail.
Flags: COLLAPSE = many old values became one; ZEROED = all became 0 or empty; SIGN = sign flip; CASE = letter case only; UNIFORM = one old→new pair; UP/DOWN = every number moved one way; GONE = elements removed, none added; UNTOUCHED = suites whose YAML the diff does not touch; PINNED = static in the harness unless UNFILTER_EXPORT=1; OPAQUE = hashes or signatures.
Full detail (every field, examples by file and block): c0d-detail.md.
Benign: python3 scripts/golden-diff-summary.py 4282a76fec^ 4282a76fec (SOL durable-nonce outbounds, 285 golden files)
Regression golden changes, by field
c3d15b5dd54c..4282a76fecfa: 285 golden files diffed in 184 suites (184 whose suite YAML this diff does not touch). 0 values changed, 609 keys or elements added, 0 removed, in 3 field groups.
Before adding goldens-reviewed, confirm every flagged row is intended. COLLAPSE, ZEROED, SIGN and CASE are how wrong behavior has been pinned before (!4140 (merged) → !4394 (merged), !4728 (merged), !4730 (merged)).
| # | Field | Changed | +/− | Files/suites | Top transitions (or values added/removed) | Flags |
|---|---|---|---|---|---|---|
| 1 | export.app_state.thorchain.sol_vault_nonces |
+184 | 184/184 | +[] ×184 | ||
| 2 | blocks[].txs[].tx.body.messages[@type=/types.MsgObservedTxIn].txs[].sol_nonce |
+267 | 87/87 | +null ×267 | ||
| 3 | blocks[].txs[].tx.body.messages[@type=/types.MsgObservedTxOut].txs[].sol_nonce |
+158 | 53/53 | +null ×158 |
Flags: COLLAPSE = many old values became one; ZEROED = all became 0 or empty; SIGN = sign flip; CASE = letter case only; UNIFORM = one old→new pair; UP/DOWN = every number moved one way; GONE = elements removed, none added; UNTOUCHED = suites whose YAML the diff does not touch; PINNED = static in the harness unless UNFILTER_EXPORT=1; OPAQUE = hashes or signatures.
What it does not cover
- It does not post an MR comment. CI has no existing token mechanism for posting MR notes (the
visual-reviewjob only records an environment; huginn posts from outside CI), so it prints to the log and saves an artifact. - It does not diff goldens that are added or deleted; it lists them. That includes goldens added with a new suite.
- Pairing is heuristic. When an array gains or loses events, some elements show as added or removed rather than changed, and generic bank events (
coin_spent,transferand so on) can appear as balanced+N −Mrows. - It flags patterns, not intent. A deliberate collapse, such as a constant set to 1, is still flagged and needs a sentence in the MR.
Follow-ups
- Post or update a single MR comment with the summary once a CI token for MR notes exists. huginn could post it instead.
- Optionally run the summary in the regression job on
EXPORT=1diffs, so authors see it before they push. - 14 refund
scheduled_outbound.coin_decimalsvalues went8→0in !4140 (merged) (row 3 above). Most of them are still0on develop. Decimals 0 probably means "chain default", but someone should confirm.