Python tests: add tests for mainnet and proposed protocol
Context
Copied from tezos/tezos!2288 (merged) as the context is the same:
We are trying to move away from
proto-proposal
and develop the protocol proposal inmaster
directly. But if we modifyproto_alpha
inmaster
, all tests that currently run onproto_alpha
would now run on the protocol being developed instead of the protocol which is currently active on Mainnet. Instead, we would probably like to run tests on both protocols.
See tezos/tezos!2303 (merged) for an overview of this effort of which this is but one MR.
This merge request duplicates most pytests by the protocol to make this possible: one set of tests, in tests_python/tests_007
target the mainnet protocol (007 = delphi). The other set, in tests_python/tests_alpha
target the protocol proposal (008 = edo = alpha).
Furthermore, the same duplication has been made for the directories
tests_python/contracts
:
There are two protocol specific directories:
- contracts{_007,}
-
contracts_007
contains michelson scripts for 007 -
contracts_alpha
contains michelson scripts for alpha - tests{_007,}
-
tests_007
contains pytest for 007 -
tests_alpha
contains pytest for alpha
I didn't see the point in duplicating the directory scripts
, that
contains utility scripts, per protocol. So there is only scripts
that works on protocol alpha (i.e. 008.)
This MR replaces tezos/tezos!2302 (closed) which was created with the wrong target branch.
To specialize the tests
In summary, this MR does the following;
- start from proto-proposal branch, thus containing tests for current protocol proposal edo
- cherry-pick !2290
- move
contracts
,tests
tocontracts_alpha,
tests_alpha` - import
contracts
,tests
and call themcontracts_007
resptests_007
from branch master - in
tests_alpha
andtests_007
: add aprotocol.py
that specific functions/constants for that protocol. notably, the functionactivate
which activatesdelphi
andalpha
fortests
andtests_008
respectively. - in
tests_alpha
andtests_007
: add acontract_paths.py
that contains the paths to the contracts incontracts_alpha
/contracts_007
respectively. - the first line of the regression logs for 007/alpha has been updated as their respective tests have been moved.
- run both respective test suites in the CI (this adds a heap of jobs)
Discussion:
Clearly this degree of duplication is not desirable.
- What to do when modifying existing tests?
- What to do when protocols are rotated?
- What happens if the testing framework must evolve in a way that is incompatible with the current protocol?
Manually testing the MR
- Check out the pipeline
- Run tests locally
cd tests_python && poetry run pytest tests_alpha tests_008/
- That's quite slow, there are now some 3000 tests... the following command timeouts slow tests and selects any marked as slow:
cd tests_python && poetry run pytest --timeout=30 -m "not slow" tests tests_008/
- verify that the contents of
tests_python/{tests,contracts}_007
is the same as the contents ofmaster:tests_python/{tests,contracts}
, modulo diffs described above - verify that the contents of
tests_python/{tests,contracts}_alpha
is the same as the contents ofproto-proposal:tests_python/{tests,contracts}
, modulo diffs described above
Checklist
- [ ] Document the interface of any function added or modified (see the coding guidelines)
- [ ] Provide automatic testing (see the testing guide).
- [ ] Add item in the Development Version
section of CHANGES.md
(only for new features and bug fixes).