Skip to content

Manifest + Tezt = easy to define local Tezt tests

Romain requested to merge nomadic-labs/tezos:romain-tezt-manifest into master

Context

The idea is to be able to define Tezt tests in libraries, in particular unit tests.

Devs would just:

  • create test files like src/lib_stdlib/tezt/test_stdlib.ml;
  • use Test.register in those files;
  • declare those tests in the manifest with the tezt function of module Manifest.

Then one could use dune build @src/lib_stdlib/runtezt to run those tests and get something like:

tezos_stdlib_test_something_tezt_exe alias src/lib_stdlib/tezt/runtezt
[13:44:53.708] [SUCCESS] (1/1) dummy test for tezos-stdlib (something)
tezos_stdlib_test_stdlib_tezt_exe alias src/lib_stdlib/tezt/runtezt
[13:44:53.719] [SUCCESS] (1/1) dummy test for tezos-stdlib

Or one can also run the test from the main Tezt executable which gathers all tests.

Each test only depends on the dependencies that are declared in the manifest + the tezt library, i.e. they can be quick to compile if they don't depend on, say, tezos-base. For unit tests that do not depend on tezt-tezos for instance, it should result in faster compilation times.

The runtezt Alias

In order not to run Tezt tests in the CI twice (once with dune runtest src/lib_X and once with dune exec tezt/tests/main.exe), Tezt tests are not added to the runtest alias. They are added to the runtezt alias instead. Which means that one cannot run dune runtest src/lib_X to run Tezt tests: one has to run dune build @src/lib_X/runtezt.

Additionally, for packages which have Tezt tests, we add dune exec runtezt {with-test} to the .opam files.

Having a different command (runtest vs build @runtezt) is not ideal, but it was the easiest way I could find to exclude Tezt tests from the CI unit tests.

Caveat

Tests in src/proto_alpha will eventually be duplicated during the snapshot, and there is some code to make sure their ~title will change when this happens. This piece of code should probably, eventually, be put somewhere where it can be shared among other protocol tests, maybe even enforced somehow.

Manually testing the MR

This MR ported src/proto_alpha/lib_protocol/test/regression. To test it:

make -C manifest
# There is a runtezt alias.
dune build @src/proto_alpha/lib_protocol/test/regression/runtezt
# You can also just run the executable (this allows to pass command-line arguments).
dune exec src/proto_alpha/lib_protocol/test/regression/main.exe
# Tests are also available from "the big Tezt executable".
# This is how the CI runs tests, to benefit from auto-balancing.
dune exec tezt/tests/main.exe -- -f test_logging.ml
# Also works from other directories.
cd src/proto_alpha/lib_protocol/test/regression
dune build @runtezt
dune exec ./main.exe

You should also check that the CI actually runs the test_logging.ml tests.

Checklist

  • Document the interface of any function added or modified (see the coding guidelines)
  • Document any change to the user interface, including configuration parameters (see node configuration)
  • Provide automatic testing (see the testing guide).
  • For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
  • Select suitable reviewers using the Reviewers field below.
  • Select as Assignee the next person who should take action on that MR
Edited by Romain

Merge request reports