Manifest + Tezt = easy to define local Tezt tests

Context

The idea is to be able to define Tezt tests in libraries, in particular unit tests.

Devs would just:

• create test files like src/lib_stdlib/tezt/test_stdlib.ml;
• use Test.register in those files;
• declare those tests in the manifest with the tezt function of module Manifest.

Then one could use dune build @src/lib_stdlib/runtezt to run those tests and get something like:

tezos_stdlib_test_something_tezt_exe alias src/lib_stdlib/tezt/runtezt
[13:44:53.708] [SUCCESS] (1/1) dummy test for tezos-stdlib (something)
tezos_stdlib_test_stdlib_tezt_exe alias src/lib_stdlib/tezt/runtezt
[13:44:53.719] [SUCCESS] (1/1) dummy test for tezos-stdlib

Or one can also run the test from the main Tezt executable which gathers all tests.

Each test only depends on the dependencies that are declared in the manifest + the tezt library, i.e. they can be quick to compile if they don't depend on, say, tezos-base. For unit tests that do not depend on tezt-tezos for instance, it should result in faster compilation times.

The runtezt Alias

In order not to run Tezt tests in the CI twice (once with dune runtest src/lib_X and once with dune exec tezt/tests/main.exe), Tezt tests are not added to the runtest alias. They are added to the runtezt alias instead. Which means that one cannot run dune runtest src/lib_X to run Tezt tests: one has to run dune build @src/lib_X/runtezt.

Additionally, for packages which have Tezt tests, we add dune exec runtezt {with-test} to the .opam files.

Having a different command (runtest vs build @runtezt) is not ideal, but it was the easiest way I could find to exclude Tezt tests from the CI unit tests.

Caveat

Tests in src/proto_alpha will eventually be duplicated during the snapshot, and there is some code to make sure their ~title will change when this happens. This piece of code should probably, eventually, be put somewhere where it can be shared among other protocol tests, maybe even enforced somehow.

Manually testing the MR

This MR ported src/proto_alpha/lib_protocol/test/regression. To test it:

make -C manifest
# There is a runtezt alias.
dune build @src/proto_alpha/lib_protocol/test/regression/runtezt
# You can also just run the executable (this allows to pass command-line arguments).
dune exec src/proto_alpha/lib_protocol/test/regression/main.exe
# Tests are also available from "the big Tezt executable".
# This is how the CI runs tests, to benefit from auto-balancing.
dune exec tezt/tests/main.exe -- -f test_logging.ml
# Also works from other directories.
cd src/proto_alpha/lib_protocol/test/regression
dune build @runtezt
dune exec ./main.exe

You should also check that the CI actually runs the test_logging.ml tests.

Checklist

• Document the interface of any function added or modified (see the coding guidelines)
• Document any change to the user interface, including configuration parameters (see node configuration)
• Provide automatic testing (see the testing guide).
• For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
• Select suitable reviewers using the Reviewers field below.
• Select as Assignee the next person who should take action on that MR

assigned to @romain.nl

added test ⚒ teststezt labels

mentioned in merge request !4918 (merged)

added 1 commit

• 7837b60a - only one lib and exe per tezt dir

Compare with previous version

added 1 commit

• 97c6a7f0 - update doc

Compare with previous version

Thank you for your contribution , I have a couple of remarks:

• There are a couple of touched files that contain nonstandard "TODO" and "FIXME" comments:
  • → main.ml
  • → manifest.ml
  • Please help stadardize these as described in the tezos documentation.
• Commit messages should match "Component: Imperative action":
  • → e92d031c: Manifest : fix opam dependencies with private libs
  • → 5cb6bc3e: demo (TO REMOVE)

---

I,  , ran this inspection on Thu, 19 May 2022 12:28:50 +0000. I usually run inspections every 6 hours and will try to edit my own comment if I have any updates. For more information see the Tezos documentation.

added 1 commit

• 1ab4b0d1 - remove hack for tezt_lib

Compare with previous version

added 1 commit

• ed327a98 - have manifest generate tezt/tests/dune to link with all other tests

Compare with previous version

changed the description

changed the description

changed the description

requested review from @sventimir

added 254 commits

• ed327a98...2e4ff41c - 248 commits from branch tezos:master
• 2786abb0 - WIP
• fdb28dc4 - only one lib and exe per tezt dir
• 22bb400f - update doc
• 4de61e66 - remove hack for tezt_lib
• 5c128321 - have manifest generate tezt/tests/dune to link with all other tests
• 38dfd775 - Manifest: do not exclude tezt/tests

Compare with previous version

added 2 commits

• 466b8bc0 - Manifest: alias runtezt for Tezt tests
• c3712c9b - Manifest: call runtezt in .opam files

Compare with previous version

changed the description

added 1 commit

• 3ebb1367 - Manifest: build @runtezt instead of runtezt, but it does not work :(

Compare with previous version

added 1 commit

• 2945da27 - ocamlformat

Compare with previous version

requested review from @hhugo1

added 179 commits

• 2945da27...abfce96e - 178 commits from branch tezos:master
• 5e2b98b2 - WIP

Compare with previous version

added 2 commits

• 87d732fb - private targets
• 62c165dd - put back runtezt

Compare with previous version

added 1 commit

• 94ea6f82 - WIP

Compare with previous version

added 37 commits

• 94ea6f82...838ee55f - 36 commits from branch tezos:master
• 05a3da71 - WIP

Compare with previous version

added 1 commit

• 7c9fd79b - WIP

Compare with previous version

added 1 commit

• 297d340d - fix circular detection

Compare with previous version

added 2 commits

• 9489498f - fix circular detection
• bd8e7d55 - WIP

Compare with previous version

added 5 commits

• 991c9808 - Manifest: fix circular detection involving private targets
• 93cdc836 - Manifest: rename tezt into tezt_lib
• 8a428d71 - Manifest: be able to specify runtest alias
• 8dc8fc6b - Manifest: be able to declare local Tezt tests
• ce94a892 - demo (TO REMOVE)

Compare with previous version

The 2 opam tests in the CI fail with:

------------------- opam_logs/log/tezos-stdlib-19497-f99f4b.out -------------------
File "src/lib_stdlib/tezt/dune", line 8, characters 2-6:
8 |   tezt
      ^^^^
Error: Library "tezt" not found.

and

------------------- opam_logs/log/tezos-store-52216-bc6cf9.out -------------------
File "src/lib_store/tezt/dune", line 10, characters 2-12:
10 |   tezt-tezos)
       ^^^^^^^^^^
Error: Library "tezt-tezos" not found.

Which I don't really understand since opam-pin.sh should have pinned it. I'll investigate.

mentioned in merge request !5319 (merged)

mentioned in merge request !5328 (merged)

!4918 (merged) is merged. Would you mind adapting it to use manifest in this MR? Otherwise, let's have a task for that.

added 110 commits

• ce94a892...582404a1 - 101 commits from branch tezos:master
• 0d171be5 - Manifest: allow private libs for a specific package
• e92d031c - Manifest : fix opam dependencies with private libs
• a9aa2ef7 - Manifest: small refactoring
• 69f45ff9 - Manifest: fix circular detection involving private targets
• 2171f61a - Manifest: rename tezt into tezt_lib
• 9a54a5d0 - Manifest: be able to specify runtest alias
• 71a6562e - Manifest: be able to declare local Tezt tests
• 854efd0f - demo (TO REMOVE)
• 25160c59 - remove assert false (TO BE PUT IN OTHER MR)

Compare with previous version

added 6 commits

• d1ca8033 - Manifest: improve checks for empty opam
• f8eddca8 - Manifest: fix circular detection involving private targets
• b7f7ac02 - Manifest: rename tezt into tezt_lib
• 9dba2fa7 - Manifest: be able to specify runtest alias
• 5c7e20dd - Manifest: be able to declare local Tezt tests
• 5cb6bc3e - demo (TO REMOVE)

Compare with previous version

added 221 commits

• 5cb6bc3e...1eb5c9c5 - 212 commits from branch tezos:master
• ecb75534 - Manifest: allow private libs for a specific package
• ca57c4c2 - Manifest : fix opam dependencies with private libs
• 2065f1ab - Manifest: small refactoring
• 6f1a478c - Manifest: improve checks for empty opam
• 50fed6d4 - Manifest: fix circular detection involving private targets
• 2c342f95 - Manifest: rename tezt into tezt_lib
• 9652d377 - Manifest: be able to specify runtest alias
• b45e75f3 - Manifest: be able to declare local Tezt tests
• 9f368a83 - demo (TO REMOVE)

Compare with previous version

added 21 commits

• 9f368a83...e7ec1380 - 11 commits from branch tezos:master
• 71a7ebae - Manifest: allow private libs for a specific package
• edf6c7f1 - Manifest : fix opam dependencies with private libs
• 34d41568 - Manifest: small refactoring
• 2b05760e - Manifest: improve checks for empty opam
• ed50f6dc - Manifest: fix circular detection involving private targets
• 5247c945 - Manifest: rename tezt into tezt_lib
• 8060d23c - Manifest: be able to specify runtest alias
• 8ae2ef78 - Manifest: be able to declare local Tezt tests
• 7e613ecf - demo (TO REMOVE)
• b452d253 - Manifest: improve ~alias and runtest_js interaction

Compare with previous version

mentioned in merge request !5419 (merged)

added 312 commits

• b452d253...5ca235b1 - 307 commits from branch tezos:master
• fc413084 - Manifest: rename tezt into tezt_lib
• 27caee93 - Manifest: be able to specify runtest alias
• 0c9400bc - Manifest: be able to declare local Tezt tests
• 2cb45cc4 - demo (TO REMOVE)
• 0054c4d1 - Manifest: improve ~alias and runtest_js interaction

Compare with previous version

added 5 commits

• f5063546 - Manifest: be able to specify runtest alias
• f343afd7 - Manifest: be able to declare local Tezt tests
• 3b791711 - Manifest: fix comment
• 2690167d - Tezt: expose Base.project_root
• 017c4e59 - Alpha/test: migrate regression tests to manifest+tezt

Compare with previous version

added 1 commit

• bd706707 - Alpha/test: migrate regression tests to manifest+tezt

Compare with previous version

changed the description

added 6 commits

• 37b43e43 - Manifest: be able to declare local Tezt tests
• d85074e8 - Manifest: fix comment
• 717ccf53 - Tezt: expose Base.project_root
• f8aa5feb - Alpha/test: migrate regression tests to manifest+tezt
• 60d37dc0 - Alpha/test: make sure snapshotting will work
• f09b565a - CI: unplug lib_protocol/test/regression from unit.yml

Compare with previous version

changed the description

changed title from WIP: manifest + Tezt = easy to define local Tezt tests to Manifest + Tezt = easy to define local Tezt tests

assigned to @hhugo1 and unassigned @romain.nl

marked this merge request as draft

added 1 commit

• 958bc16d - Scripts: fix snapshot_alpha.sh for files with spaces

Compare with previous version