Manifezt: Manifest + Tezt

See milestone: %Conditional Octez tests

What

This MR introduces a script, manifezt.sh, which runs Tezt with an additional parameter to restrict its set of tests to those that actually need to be run according to the changes between master and the current working directory. In other words, running manifezt.sh --list is the same as tezt --list, and running manifezt.sh -j 4 -v is the same as tezt -j 4 -v, except that only relevant tests are listed / run.

Why

In the future, we plan to use this script in the CI. This should significantly reduce the CI time and resources needed for some MRs, in particular layer 2 MRs. For instance, an MR that only modifies the rollup node only needs to run tests that run the rollup node. Of course, if your MR modifies the L1 node, you will still have to run most tests (but not all).

How

manifezt.sh runs git diff to list modified files. Then it calls the manifest with this list of files. The manifest computes a TSL expression (TSL is the Test Selection Language of Tezt). The script then simply runs Tezt with this TSL expression. The hard part is obviously the code in the manifest, which has to figure out what tests need to be run given a list of changes.

The manifest has access to all OCaml dependency relationships. So it can deduce that, for instance, a change in lib_store means that all tests in lib_store/test need to be run. The manifest also knows that modifying lib_store modifies the node and the DAL node. It can deduce that tests that run the node or the DAL node need to be run as well. Those are tests with tag node and dal_node.

To know which Tezt tags to run given a list of modified executables, the manifest reads the following file:

tezt/lib_wrapper/expected/tezt_wrapper.ml/meta- list runtime dependencies.out

This file is generated by a regression test in tezt_wrapper.ml. Since Tezt_wrapper knows all Uses.t that were declared in Tezt, it can list them in this file. By doing that in a regression test, we also make sure that the file is always up-to-date.

What's Missing

This MR is missing some subtle features that I plan to do in another MR:

• if you modify tezt/lib_tezos, the script does not know that all integration tests need to be run;
• if you modify a test in tezt/tests, the script does not know that this test needs to be run;
• it does not handle some deps annotations (dependencies on files, globs, etc.) from the manifest (done in romain-manifezt-2).

Those are identified with a TODO comment. I know, it does not follow the rules, I didn't create an issue, but it's part of the milestone anyway.

We can still merge though, because the changes do not impact everyday use at all, and the script can still be useful even if not complete.

Manually testing the MR

Modify some source files, then run:

manifest/manifezt.sh --list

See if the list of tests makes sense for the changes you made.

You can also run:

make -C manifest manifest
manifest/manifest --manifezt src/bin_node/main.ml

to see the TSL expression that the manifest comes up with when you modify src/bin_node/main.ml. You can pass multiple files.

When running manifezt.sh, if you do not want to compare with master, you can specify the ORIGIN environment variable. For instance, to compare with HEAD:

ORIGIN=HEAD manifest/manifezt.sh --list

You can also activate verbose mode with MANIFEZT_DEBUG=true to understand how the manifest came up with the TSL expression:

make -C manifest manifest
MANIFEZT_DEBUG=true manifest/manifest --manifezt src/bin_node/main.ml
MANIFEZT_DEBUG=true ORIGIN=HEAD manifest/manifezt.sh --list

Checklist

• Document the interface of any function added or modified (see the coding guidelines)
• Document any change to the user interface, including configuration parameters (see node configuration)
• Provide automatic testing (see the testing guide).
• For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
• Select suitable reviewers using the Reviewers field below.
• Select as Assignee the next person who should take action on that MR