3.8 protectedRelease Release 3.8
Release 3.8 - Maintenance release (bug fixes, improved tests)
Download source code
3.7 protectedRelease Release 3.7
Release 3.7 - Verbose logging only verbosely logs messages of TestCov, not of other subcomponents. Especially matplotlib debug messages are not displayed anymore when command-line flag '--verbose' is used. This reduces clutter in the verbose output. - The log prefix 'DEBUG:root' changed to 'DEBUG:suite_validation'. - Nicer-looking progress display: Log messages and the progress display now work nicely together and do not intermingle, and the progress display shows the currently executed and total number of tests. Example: "⏳ Executing tests 20/227"
Version 3.6 Added - Report total number of tests found in a given test suite. We previously only reported the number of executed tests, but it may be interesting to see whether all identified tests were executed. Fixed - A keyboard interrupt (SIGINT, usually Ctrl+C) does not display an exception anymore, but gracefully terminates. On an interrupt during test-suite execution, execution will stop and result files will be produced. It is then possible to abort the generation of result files by interrupting again. - Get correct coverage measurement for single-line functions that call __assert_fail. A gcov_dump to dump recorded coverage was missing for these cases.
v3.5 protectedRelease Version 3.5
Version 3.5 Changed - Improved speed and supported language features for test-goal instrumentation in TestCov. We've replaced the Python-based implementation of adding test goals to the program with an implementation based on clang libtooling. This allows us to support the full range of C language features and GCC extensions. Before, we had to abort if unsupported features were used; most notably variadic arguments, but also GCC extensions that we did not remove explicitly. Fixed - Removed a bug that made TestCov fail if preprocessed programs contained method declarations of builtin methods with the Star operator '*' right before the method name (e.g., '*malloc'). Such preprocessed programs are now fully supported.
v3.4 protectedRelease Version 3.4
Version 3.4 Added - Always add calls to __gcov_flush before abort and __assert_fail calls. Before, this only happened for coverage goals "branch coverage" and "function-call coverage". This means that coverage measurements were not stored for condition and line coverage if an assert or abort was hit. - TestCov and its dependencies can now be installed with `pip install .` Changed - Changed __gcov_dump to __gcov_flush (__gcov_dump does not exist anymore in GCC 11). - Support new naming scheme and directories for GCDA and GCOV files starting with GCC 11. - During code transformation, TestCov now always removes preprocessor comments so that the original file name is not used in GCOV output (starting with GCC 11), but the name of the transformed file. This avoids user confusion in the output. - Removed unnecessary semicolons after compiler directives in test harness. - Improved error messages for missing files and on gcov and lcov errors. - Cosmetic change: Call gcc with `--coverage` instead of `-fprofile-arcs -ftest-coverage`. - For testing, switched from nosetest to pytest - Store all metadata in setup.cfg instead of setup.py. (easier to read and can contain more information about other tools) - Minor code improvements
v3.3 protectedRelease Version 3.3
Version 3.3: Bug squashing in program instrumentation
Version 3.2: Reduce number of labels injected into program code for measuring branch coverage
Version 3.1: Vastly improved program instrumentation for coverage and speed improvements in coverage computation
testcomp20 protectedRelease Test-Comp 2020 release
Version used in Test-Comp'20
Version 3.0: Rename and new features * Rename test-suite validator to TestCov (`bin/testsuite-validator` is now `bin/testcov`) * Add additional test-suite reduction strategy based on count distance between coverages * Merge the two plots created for accumulated and individual coverage into single plot
Version 2 of testsuite-validator * Runs tests in a container through runexec. You can use `--memlimit MEM` and `--cpu-cores #CORES` to specify these limits (e.g.: `--memlimit 2GB --cpu-cores 2`) This requires access to cgroups (cf. https://github.com/sosy-lab/benchexec/blob/master/doc/INSTALL.md#setting-up-cgroups) If it is not possible for you to provide this access, use argument `--no-runexec`. With this parameter, testsuite-validator will only use containerexec, but no cgroups. * Creates not only coverage information for line and branch coverage, but also for condition coverage. Previously, the number of evaluated conditions was shown ('branches executed'), but not the conditions taken. 'Branches executed' is not displayed anymore because we don't see any usage for it. * Coverage goals are provided through property files in Test-Comp format (https://test-comp.sosy-lab.org/2019/rules.php). If the goal is 'coverage-error', test-suite execution stops after a covering test is found. Argument `--stop-after-success` is replaced by this. * The coverage to display in coverage statistics is chosen according to the coverage goal provided. * Creates additional coverage statistics for individual tests and plots for this data. To turn any of these off, use argument `--no-sequence-file`, `--no-individual-test-coverage`, and `--no-plots`. * Creates a reduced test suite not only for coverage goal coverage-error, but also for line, branch and condition coverage. To disable this, use argument `--no-create-reduced-suite`
First standalone version of tbf-testsuite-validator. Identical in features to v1.2, but with small adjustments to be independent of other test-format tools. Previous project tbf-testsuite-validator belonged to: https://gitlab.com/sosy-lab/software/test-format/