Version 2 of testsuite-validator
* Runs tests in a container through runexec.
You can use `--memlimit MEM` and `--cpu-cores #CORES`
to specify these limits (e.g.: `--memlimit 2GB --cpu-cores 2`)
This requires access to cgroups (cf.
https://github.com/sosy-lab/benchexec/blob/master/doc/INSTALL.md#setting-up-cgroups)
If it is not possible for you to provide this access,
use argument `--no-runexec`. With this parameter,
testsuite-validator will only use containerexec,
but no cgroups.
* Creates not only coverage information for line and branch coverage,
but also for condition coverage.
Previously, the number of evaluated conditions
was shown ('branches executed'), but not the conditions taken.
'Branches executed' is not displayed anymore because we don't see
any usage for it.
* Coverage goals are provided through property files in Test-Comp
format (https://test-comp.sosy-lab.org/2019/rules.php).
If the goal is 'coverage-error', test-suite execution stops after
a covering test is found.
Argument `--stop-after-success` is replaced by this.
* The coverage to display in coverage statistics is chosen
according to the coverage goal provided.
* Creates additional coverage statistics for individual tests
and plots for this data. To turn any of these off,
use argument `--no-sequence-file`, `--no-individual-test-coverage`,
and `--no-plots`.
* Creates a reduced test suite not only for coverage goal coverage-error,
but also for line, branch and condition coverage.
To disable this, use argument `--no-create-reduced-suite`