Test Selection Language (TSL)

Description / Motivation

There are predicates that one cannot express to select tests from the command-line. For instance, one cannot say "select all tests from file f.ml, and also tests that have tag toto".

With tezos/tezos!10639 (closed) we will likely need to be able to express this kind of predicates.

This MR introduces a general Test Selection Language (TSL). The above example can now be done with:

tezt 'file = f.ml || toto'

This MR doesn't add tests yet. Testing the parser and pretty-printer should be done, but for now I'm just testing the waters.

Here is the specification of TSL, extracted from --help:

    You can also specify more complex predicates using the Test Selection
    Language (TSL). For instance:

    _build/default/test/unix/main.exe 'file = bootstrap.ml || bake && /rpc'

    runs all tests defined in bootstrap.ml, as well as tests (possibly from
    other files) with tag 'bake' but without tag 'rpc'. This example cannot be
    expressed without TSL.

    Passing multiple predicates is equivalent to passing their conjunction as a
    single predicate. For instance, passing the two command-line arguments 'a ||
    b' and 'c || d' is equivalent to passing the single argument '(a || b) && (c
    || d)'.

    TSL allows the following expressions, where STRING denotes a string and EXPR
    denotes a TSL expression:
    - 'true': always true;
    - 'false': always false;
    - 'EXPR && EXPR': conjunction;
    - 'EXPR || EXPR': disjunction;
    - 'not EXPR': negation;
    - 'STRING': test has tag STRING;
    - '/STRING': same as 'not STRING' (STRING must not be quoted);
    - 'file = STRING': test file is STRING;
    - 'file <> STRING': same as 'not (file = STRING)';
    - 'title = STRING': test title is STRING;
    - 'title <> STRING': same as 'not (title = STRING)';
    - 'file =~ STRING': test file matches the regular expression STRING;
    - 'file =~! STRING': same as 'not (file =~ STRING)';
    - 'title =~ STRING': test title matches the regular expression STRING;
    - 'title =~! STRING': same as 'not (title =~ STRING)';
    - '(EXPR)': same as EXPR.

    'not' has higher precedence than '&&' which has higher precedence than '||'.

    TSL strings need to be quoted using double quotes '"' unless they only
    contain characters 'a-zA-Z0-9_-./' and do not start with a slash '/'. Double
    quotes '"' and backslashes '\' need to be escaped using backslashes '\'.

    Note that 'file = STRING' is not equivalent to '--file STRING'. Indeed,
    '--file a.ml' selects 'x/a.ml', while 'file = a.ml' does not.

FAQ

Why not enforce quotes for strings?

For backward compatibility: tag needs to be a valid expression.

Bare words are also a usual features of shells anyway.

How to Test the MR Manually

$ dune exec test/unix/main.exe -- '((check && /not_like) || seed || (file =~ "/test_process.ml"))' --list
+---------------------------+--------------------------------+---------------+
|           FILE            |             TITLE              |     TAGS      |
+---------------------------+--------------------------------+---------------+
| test/common/tests.ml      | fixed seed                     | seed, fixed   |
| test/common/tests.ml      | random seed                    | seed, random  |
| test/common/test_check.ml | Check.(=)                      | check, eq     |
| test/common/test_check.ml | Check.(<>)                     | check, neq    |
| test/common/test_check.ml | Check.(<)                      | check, lt     |
| test/common/test_check.ml | Check.(<=)                     | check, le     |
| test/common/test_check.ml | Check.(>)                      | check, gt     |
| test/common/test_check.ml | Check.(>=)                     | check, ge     |
| test/common/test_check.ml | Check.(=~)                     | check, like   |
| test/unix/test_process.ml | Process.terminate with timeout | process, kill |
+---------------------------+--------------------------------+---------------+

Checklist

• Update CHANGES.md. No need to document changes to documentation and tests.
• Make sure all new values, types etc. are documented in .mli files and that the generated documentation looks ok.
• Add tests in test/, if relevant.