Rehearse upgrades and store migrations on mainnet state before tagging

🔎 Visual review: building it for 859df963; the link appears here.

What and why

Two mainnet halts came at upgrade time, and only real mainnet state would have shown them:

  • !4029 (merged), TCY store migration: Migrate4to5 sent total-minus-claims to tcy_claim and then paid the treasury from tcy_claim. The transfer failed with insufficient funds in FinalizeBlock at the upgrade height. CI never ran it, because it lives in the mainnet-tagged migrations file and CI built only mocknet.
  • !4357 (merged), 3.13.0 upgrade panic: the shared standard upgrade still had x/scheduler in StoreUpgrades.Added from 3.12. The store loader re-added an existing store, and the node panicked at the upgrade height.

The retrospective (bug_class upgrade-or-migration, why_missed upgrade-path-untested) has six more of these. The closest one is !4310 (merged): ConsensusVersion was bumped to 10 and Migrate9to10 added, but the 9→10 step was never registered, and rc1 to rc3 shipped like that. In all of these, nothing in CI ran the upgrade against prior-version state.

This MR runs that upgrade path in two layers. Both use one driver, app/rehearsal:

  1. Write upgrade-info.json, so the branch's store loader applies the upgrade's StoreUpgrades.
  2. Build the branch THORChainApp on the mainnet db.
  3. Schedule the upgrade for the next height.
  4. Finalize and commit that block and N more by calling ABCI directly. No CometBFT and no validator set are needed.

The driver fails on any of these:

  • a store-loader panic;
  • a handler error;
  • an error logged while the handler ran (soft-skipped run/apply migrations, undeclared LUVI drops);
  • a module left at the wrong consensus version;
  • StoreUpgrades that do not match mainnet's store set;
  • an x/thorchain invariant (keeper/v1/invariants.go) that held before the upgrade and breaks after the handler or after the last block.

On develop between releases, every registered upgrade has already run on mainnet. The driver then rehearses standard.NewUpgrade under the next minor version, which is today's 3.22.0.

Layer 1, every MR on the upgrade path. This is CI job test-upgrade-rehearsal, which runs make test-upgrade-rehearsal (-tags mainnet) in about 10 s.

  • Fixture: app/rehearsal/testdata/mainnet-slice.json.gz is 4.2 MB, taken at height 28,697,916 on THORNode 3.21.4. tools/mainnet-slice wrote it from the public RPC with abci_query /store/<name>/subspace, retrying until every read landed on one height.
  • What the slice holds: pools, vaults, nodes, mimirs, network, TCY, LPs, swap queues, RUNEPool, module accounts and balances, bank supply, the whole upgrade, consensus and params stores, and mainnet's store list. It leaves out history, user balances, THORNames and contract code.
  • Refresh: make refresh-mainnet-slice. MAINNET_SLICE_RPC can name a keyed RPC; the tool never prints or stores it.
  • When the job runs: on MRs whose changes match rules:changes, which covers app/upgrades*, app/app.go, x/thorchain/migrations*.go, migration_*.go, module.go, luvi_migration.go, the invariants, the version file and go.mod. It always runs on develop, mainnet, stagenet, chainnet, release* branches, tags and schedules.
  • Live variant: in schedules, test-upgrade-rehearsal-live fetches a fresh slice, rehearses on it, and keeps the slice as an artifact.

Layer 2, nightly and before tagging, on hedwig. This is CI job upgrade-rehearsal-snapshot (tags: [hedwig], resource_group, 3 h timeout), which runs scripts/upgrade-rehearsal/run.sh. The script does four things:

  1. Finds the newest mainnet-pruned snapshot from the Liquify snapshot API that node-launcher's restore-external-snapshot uses. Today that snapshot is 35.6 GB, published daily, with its sha256.
  2. Keeps one verified tarball in /upgrade-rehearsal/snapshots.
  3. Extracts only application.db and wasm, not the block store.
  4. Runs tools/upgrade-rehearsal (-tags mainnet) for the upgrade block plus 20 blocks.

The job is off until UPGRADE_REHEARSAL_SNAPSHOT=1 is set; with it set, it runs in schedules, on vX.Y.Z-rcN and vX.Y.Z tags, on pushes to mainnet and stagenet (the mainnet branch is pushed before the release tag), and on web or API pipelines. Set it as a project CI variable once hedwig is ready (step 5 below), so release-tag pipelines do not fail on an unprepared runner.

What the released binary does in layer 2. The snapshot is the state the released binary committed. The rehearsal schedules the upgrade at the snapshot height + 1, so no pre-upgrade blocks run, and the released binary is not started. Running it for blocks H+1..H+N-1 first would need a single-validator fork. The SDK's in-place-testnet does not apply here, because THORChain sets the validator set from x/thorchain, not x/staking.

Validated on history

I exported the buggy parents and the fixes with git archive into scratch dirs. I copied app/rehearsal into each tree and rewound today's slice to the release's era: x/thorchain consensus version, applied-upgrade list, and the era's store set (adding authz, dropping scheduler where needed). Builds used GOTOOLCHAIN=go1.24.5, because sonic v1.14.1 in those trees does not build on Go 1.26.

Tree Era Result
bfdcee2a10^ (!4029 (merged) bug) 3.5.0, thorchain 4→5 FAIL: upgrade block failed: spendable balance 2815587674405905tcy is smaller than 20660654128874864tcy: insufficient funds. This is the mainnet failure.
bfdcee2a10 (!4029 (merged) fix) same PASS. The TCY mint, claims, treasury and slash refunds run.
af38755f73^ (!4357 (merged) bug) 3.13.0, thorchain 10→11 FAIL: static check store "scheduler" is in 3.13.0 StoreUpgrades.Added but mainnet already has it, plus the real loader panic initial version set to 28697754, but found earlier version 28697753.
af38755f73 (!4357 (merged) fix) same PASS
759aec3620^ (!4310 (merged) bug) 3.12.0, thorchain 9→10 FAIL: no migration found for module thorchain from version 9 to version 10

Caveat: the state is from November 2026, not from each era. Archive nodes for April 2025 are gone (Liquify's archive too). The !4029 (merged) failure does not depend on state: tcy_claim today holds 24.8M TCY, far short of 206.6M. The !4357 (merged) and !4310 (merged) failures depend only on the store set and the version map, and both of those were rewound.

How it was tested

  • go test -tags mocknet ./app/rehearsal ./tools/mainnet-slice passes, with 85.7% coverage of app/rehearsal. The unit tests rehearse a synthetic chain and assert the driver catches each of these: a re-added store (!4357 (merged)), a TCY-style transfer without funds (!4029 (merged)), a registration gap (!4310 (merged)), a broken invariant, and a swallowed migration error. A clean pending migration passes.
  • go test -tags mainnet ./app/rehearsal and make test-upgrade-rehearsal pass. On develop the rehearsal runs 3.22.0 with thorchain 20->21, so the mainnet Migrate20to21 runs on real balances. It reimburses tcso 11,768.65 RUNE from the Reserve, and no invariant breaks.
  • tools/mainnet-slice was run against the public RPC four times.
  • I ran the layer 2 path end to end on local disk, in two ways:
    • tools/upgrade-rehearsal -init-from-slice on a goleveldb home;
    • run.sh with SNAPSHOT_TARBALL set to a tarball laid out like a snapshot (data/application.db, data/wasm/cache, data/blockstore.db). That covered extract, skip, build, run and report.
  • The Liquify listing endpoint was queried (metadata only).
  • TMPDIR=$(mktemp -d) make lint-ci is clean, glab ci lint says the YAML is valid, and scripts/test-quality-inventory.py --check-selection passes with the new ./app/rehearsal mainnet selection.
  • make generate was not run, because no proto, openapi or go:generate source changed.

Not executed: the 35.6 GB snapshot download, the signed download-URL request, and any run on hedwig.

What it does not catch

  • Bugs that need state outside the slice (layer 1 only): txouts, observed txs, THORNames, user balances and contract code. Layer 2 has all of these. !4179 (merged) (7to8 failed on a stagenet network fee with zero TransactionSize) is caught only if that state exists at rehearsal time.
  • Wrong results that break no invariant and log nothing. !3831 (merged) (min-join version not updated) and the GG20 gate versions (!5291 (merged), !5327 (merged)) fall here.
  • Bifrost-side upgrade bugs. !4128 (merged) (keyshare file format) and !3748 (merged) (network fee reporting during a rolling upgrade) are out of scope.
  • Multi-validator effects: non-determinism, gas_used or app-hash divergence between nodes, and validator-set changes. This is a single process.
  • Patch releases. The rehearsal always schedules an upgrade, so it does not exercise the patch-release check that the binary still has a handler for the network's last upgrade.
  • Wasm contract migrations in layer 1. The slice holds no contract code.

Runner-side steps for hedwig (not done here)

  1. On hedwig-ci, create /srv/ci-data/upgrade-rehearsal. Add "/srv/ci-data/upgrade-rehearsal:/upgrade-rehearsal" to the volumes of runner 56849061, the main hedwig registration. The job fails at once with this hint if the mount is missing.
  2. Disk: about 150 GB free on /srv/ci-data. That is 36 GB for the cached tarball and up to about 90 GB for extraction while the job runs.
  3. Timeout: the project and runner maximum job timeout must allow 3 h.
  4. Memory: the job container's 24 GiB limit was not measured against full mainnet state. If the job is OOM-killed, raise the limit or move the job.
  5. Set UPGRADE_REHEARSAL_SNAPSHOT=1 as a project CI variable (this switches the job on for schedules, release tags and the mainnet/stagenet branches) once steps 1-4 are done.
  6. Egress to snapshot-api.liquify.com and its signed download host, which today is the DE region.

Follow-ups

  • Run layer 2 once on hedwig after steps 1 to 5. Record the db size, memory and duration, and tune UPGRADE_REHEARSAL_BLOCKS and the timeout.
  • A patch-release mode: one block with no plan, to check the wrong app version downgrade guard.
  • Run the released binary before the swap, via a single-validator fork tool for THORChain.
  • Refresh the slice after each mainnet upgrade, or automate the refresh as an MR from the live job's artifact.
  • The slice shows asgard already insolvent: 932.77 RUNE before any upgrade. This could be a slice artifact or real; it was not investigated.

🤖 Generated with Claude Code

Edited by Huginn Ai

Merge request reports

Loading
Loading