Would it make sense to use Tezt for unit tests?
Tezt was originally marketed as an integration test framework/library. But in reality nothing prevents us from using it for unit tests as well. After all, if we ignore the scope of the tests, technically an integration test is a unit test that launches external processes. So in this issue I'd like to discuss whether it would make sense to use Tezt for unit tests: what would we gain? What would we lose?
Tezt
Tezt is:
- (A) a library to register and run tests (modules
Cli
,Log
andTest
)- that can be selected individually for running
- that displays a report
- (B) a library to run external processes (modules
Process
andTemp
)- well-integrated in (A)
- with temporary files to clean up (or not, depending on command-line options)
- (C) a library to run Tezos processes (
lib_tezos
modules) - (D) (not yet, but there are discussions to add this since it is easy) some functions for common case assertions such as "fail the test with a nice error message if
a <> b
"
Alcotest
I'm not familiar enough with Alcotest to know for sure but I think it at least handles (A) and (D). If that is all it provides, it looks like Tezt could easily replace Alcotest. But I'm sure that there are killer features of Alcotest that we would miss. So we need to investigate which features they are and whether it is easy to integrate them to Tezt or to re-implement them.
QCheck
QCheck is maybe (A) and (D) (unless those are delegated to Alcotest?), and adds:
- (E) the ability to generate test cases automatically in a smart way (e.g. with shrinking to produce minimal counter-examples)
It would not make sense to re-implement (E) in Tezt: this is a complicated endeavor. So we should investigate if QCheck can be integrated in Tezt. Even if we do not want to port unit tests to Tezt, it could be useful to generate test cases automatically for integration tests. The main potential issue is that if QCheck does (A) and (D), and if its version of (A) and (D) cannot easily be decoupled from (E), then we would have a conflict with the (A) and (D) of Tezt. So we need to check whether (E) is coupled with (A) and (D) in QCheck, and if it is, how easy it would be to decouple it (by contributing to QCheck).
Benefits
So what would we gain by using Tezt in the first place?
- the ability to use
--suggest-jobs
(!2679 (merged)) to balance CI jobs - JUnit XML reports (!2776 (merged)) (!2841 (merged) seems to imply that Alcotest does not generate them)
- the ability to use
--loop
to detect flaky tests (maybe Alcotest already has that?) - nice log integration with the CI (all of these can probably be done with Alcotest if we configure it properly though?)
- only show the report in the job output (having the full log is too verbose but also the CI is limited in the size of this output: the job fails if it is too long)
- and yet still have the logs in the job output if a test fails
- and also have the full logs as an artefact in case something fails
- a uniform way to write tests in Tezos
- a uniform command-line interface to run tests (to select only tests with a given title, from a given file, or with some given tags for instance; but also with
--time
to get a report with the time each test took, etc.) - a uniform output for tests
- the ability to take a unit test and easily add integration test features to it if needed, instead of having to convert the test to an entirely different framework
Conclusion
As the author of Tezt I am clearly biased here obviously. So I need your help to know:
- why this wouldn't work;
- what are the features that we would miss from Alcotest;
- whether the things we would gain are actually useful for you;
so that we can decide if it is worth it to start using Tezt for unit tests. We of course don't have to convert existing tests, at least initially. We can write new tests with Tezt as a proof of concept first.