Meta-issue: Improve test suite

This is a meta-issue to organize work around improving the usability of the test suite. Several problems have been identified:

• It is hard for users to understand what a test is doing, why it is failing
• It is hard for users to understand what test is running, and how to rerun it
• It is hard for users to parse the output of tests
• There are differences between local and runs in the CI

These problems are aggravated by the plethora of testing frameworks used for tezos and the ensuing lack of coherency.

Propositions

Logging

Reporting

Currently:: Some frameworks are quite chatty (alcotest), whereas others apply output capture (pytest). Whereas the former write all output (when not captured) to stdout, flextesa stores output in an directory tree, where different files correspond to different commands or processes.

Propositions:

1. All frameworks shall have two modes: a verbose that displays all output and a silent mode that hides output.
2. All frameworks shall have a facility for writing all output to a log file.

Option 1.1 A third mode that hides output unless for failing tests.

Notes

For alcotest, see this pull request.

Integration test log format

Integration tests run commands and start processes. Logging can contain the commands ran, the output received, logging information from the processes.

Proposition

1. Frameworks should (have the option of) interleaving output of integration tests in one file
2. Interleaved output should (graphically) separate commands and their output, and the output of different processes should be visually distinguished.

Debugging info in unit tests

Some unit tests (notably alcotests) print debugging info to stdout. This should be removed.

Re-running tests

Currently: When a test fails, it can be hard to figure out how to rerun the failing test in isolation. dune gives a hint on the command run, but the one dune job can execute a whole suite of instructions.

Option 1: Framework should give instruction how to re-run the exact test that failed.

Option 2: The documentation of each framework should give instruction how to re-run the exact test that failed.

Understanding tests

Currently: When a test fails, it may be hard to understand what a test is doing and why it fails.

Option 1: Adding a logging system to framework, so that the test writer can output the intention of commands along with the command.

Option 2: It should be easy to find the source code of the test, and the test should be written in a self-documenting way (naming, assertions, comments, docstrings, etc.) The forthcoming testing guide should contain these points and give examples of a good standard to follow for each framework.

Local and CI testing

Currently: The tests ran in the CI and the tests run locally by make test are not the same.

This is discussed more in detail at https://gitlab.com/nomadic-labs/tezos/issues/29 in the entry for 2020-01-30. No specific proposition for the moment, RFC.

Various

Various issues to improve the test suite:

• When tezos-client / tezos-node says "unrecognized command", also print the command.
• Add environment variable that disables printing of help when unrecognized command, or alternatively do not show the help after each unrecognized command
• Make it easier to debug failing regression tests. See nomadic-labs/pytest-regtest!1 (closed)
• Improve the system for retries/polling between client / node. For instance, if a polling timesout, output the type of polling and the variables that must change to increase the timeout or the number of retries.
• Investigate parallelism and other techniques for speeding up the test suite.

Request for comments: @romain.nl @igarnier garnier @rafoo_ @bidinger