Skip to content

[add] Regression Test Framework #check-lint-warning

Ursa (9R) requested to merge ursa/regression into develop

The goal of this change is to create a pattern that allows us to significantly increase test coverage of both common and boundary conditions to achieve a high degree of confidence that ongoing fixes and features do not unexpectedly change any existing behaviors.

The regression testing framework simulates a DSL-like sequence of transactions and endpoint checks defined in YAML. While unit tests are more flexible and smoke tests cover broader interactions between Bifrost and Thornode, neither are trivial to write and reason around since most of us operate on a mental model of transactions and endpoints. Usage of the framework is documented in more detail in test/regression/README.md and there are ongoing examples of tests being added in test/regression/suites/.

In addition to explicit checks in tests, all cases compare the exported state to the last known export - this provides an additional safety mechanism to ensure there were no values changed when an explicit check was forgotten.

Earmarks:

  • Cleanup dead wait in smoke test CI target
  • Mocknet dependency fixes
  • Fix THOR_BLOCK_TIME mapping so that CI smoke tests correctly run with shorter block time
  • Reset default propose and prevote deltas back to 500ms (set to 1ms following consensus failure a few months ago)
  • Separate const defined in x/thorchain using the regtest build tag to set minimum ILP blocks to enable testing of ILP cutoff
  • Added a recover in the external handler func because currently panics in message handling silently fail

Work in Progress:

  • Output coverage of all non-archived code paths.

Future Work:

  • Current implementation only interacts with Thornode and simulates observations, but we intend in the future to extend the operations types to support all mocknet chains so that this framework can replace smoke tests entirely.
Edited by Ursa (9R)

Merge request reports