WIP: Add stack trace regression test
Add stack trace regression tests. The basic idea is to commit the expected output of certain commands (i.e.
tezos-client run script <contract> on storage <storage> and input <input> --trace-stack) and to add pytests that verifies that re-running the command returns same output as the committed result.
- Is currently limited to verifying the stack traces of the tests specified in
test_contract_opcodes.pyin the method test_contract_input_output
How to test this MR
Follow the instructions below to install the required dependencies, and then run
Follows the excerpt from the developer documentation included in this MR:
Stack trace regression testing
Stack trace regression testing is a coarse-grained testing method for
detecting unintended changes in the behavior of the Michelson
interpreter. It consists of running a suite of test commands using the
tezos-client and verifying that the output has not changed
with respect to a a log file stored in the source tree.
How to install
Running regressions tests requires a modified version of
pytest-regtest that is included in the tezos source tree. To install
pip3 install -r < requirements.txt in
The modifications fix a bug in pytest-regtest. Tests with long parameterizations would create log files whose length exceed the OS path length limit. The patch replaces explicit parameters in log files with a hash after a certain length.
How to run
Contract opcodes regression tests
The regression tests for contract opcodes verifies that the output of each execution of the tezos-client is the same (modulo hashes, timestamps, counters, etc ) as the stored log for each test.
These tests are included in normal test execution. I.e. to run these tests:
How to update regression test logs
The test logs are stored in tests_python/tests/_regtest_outputs/. If the log needs to be updated, pass '--regtest-reset' to pytest.py:
The resulting changes should be committed.
Architecture (fixtures etc)
Regressions tests are written by employing the regtest fixture
provided by pytest-regtest. To run regtests on client, a version of
the client fixture called
client_regtest is included that intercepts
all commands passed to the client and verifies the output against the
The output of the client might differ slightly from test one run to
another, for instance due to timestamps. A specialized fixture
client_regtest_scrubbed applies a series of conversions to the
output. For example, a timestamp such as 2019-09-23T10:59:00Z is
replaced by [TIMESTAMP]. These are defined in the function
client_output_converter of conftest.py.
tl;dr: do not mix the client fixture and the client_regtest(_scrubbed) in the same test class.
The client_regtest fixture uses a specialized version of the sandbox
fixture under the hood
client_regtest_bis. This fixture has a class
scope. On the other hand, regression test must be function-level
fixtures, as each function (test) should have one output file.
sandbox (class) > client (class)
sandbox (class) > client_regtest_bis (class) > client_regtest (function) >
It is client resp. client_regtest_bis that adds a node to the sandbox. If a client fixture and a client_regtest fixture is used in the same test_class, then two nodes will be added to the sandbox. Their interference might cause unintended consequence disturbing the tests.