[add] Regression Test Framework #check-lint-warning
The goal of this change is to create a pattern that allows us to significantly increase test coverage of both common and boundary conditions to achieve a high degree of confidence that ongoing fixes and features do not unexpectedly change any existing behaviors.
The regression testing framework simulates a DSL-like sequence of transactions and endpoint checks defined in YAML. While unit tests are more flexible and smoke tests cover broader interactions between Bifrost and Thornode, neither are trivial to write and reason around since most of us operate on a mental model of transactions and endpoints. Usage of the framework is documented in more detail in test/regression/README.md
and there are ongoing examples of tests being added in test/regression/suites/
.
In addition to explicit checks in tests, all cases compare the exported state to the last known export - this provides an additional safety mechanism to ensure there were no values changed when an explicit check was forgotten.
Earmarks:
- Cleanup dead wait in smoke test CI target
- Mocknet dependency fixes
- Fix
THOR_BLOCK_TIME
mapping so that CI smoke tests correctly run with shorter block time - Reset default propose and prevote deltas back to 500ms (set to 1ms following consensus failure a few months ago)
- Separate const defined in
x/thorchain
using theregtest
build tag to set minimum ILP blocks to enable testing of ILP cutoff - Added a
recover
in the external handler func because currently panics in message handling silently fail
Work in Progress:
-
Output coverage of all non-archived code paths.
Future Work:
- Current implementation only interacts with Thornode and simulates observations, but we intend in the future to extend the operations types to support all mocknet chains so that this framework can replace smoke tests entirely.