README.md 4.57 KB
Newer Older
1 2 3 4 5 6 7 8 9 10
<!--
This file is part of TestCov,
a robust test executor with reliable coverage measurement:
https://gitlab.com/sosy-lab/software/test-suite-validator/

SPDX-FileCopyrightText: 2019-2020 Dirk Beyer <https://www.sosy-lab.org>

SPDX-License-Identifier: Apache-2.0
-->

11
# TestCov
Thomas Lemberger's avatar
Thomas Lemberger committed
12

13 14
[![Apache 2.0 License](https://img.shields.io/badge/license-Apache--2-brightgreen.svg?style=flat)](https://www.apache.org/licenses/LICENSE-2.0)

15 16 17 18 19 20 21
TestCov is a robust test-suite executor for C programs.
It uses Linux containers and namespaces to ensure robust and repeatable execution and coverage measurement of test suites.

For coverage computation, TestCov uses [gcov](https://gcc.gnu.org/onlinedocs/gcc/Gcov.html)
and [lcov](https://github.com/linux-test-project/lcov).
For containerization, TestCov uses parts of [BenchExec](https://github.com/sosy-lab/benchexec/).

22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
## Requirements

* Python >= 3.6
* gcc >= 8.0
* lcov >= 1.13

The following requirements are automatically installed by `setup.py` upon installation,
but can also be installed with `pip install -r requirements.txt`:
* lxml >= 4.0
* numpy >= 1.15
* BenchExec >= 1.20
* pycparser >= 2.19

Older versions of GCC can be used, but may mistakenly mark the last else-branch of a program
as covered, even it if wasn't. We thus recommend to use gcc version 8.0 or later.

Optional, for plotting (if not available, run `testcov` with argument `--no-plots`):
* matplotlib >= 3.1.0

For development, we use the [`black`](https://github.com/python/black) formatter,
[`pylint`](https://www.pylint.org/)
and [`nosetest`](https://nose.readthedocs.io/en/latest/).

## Installation

47
Because of additionally shipped libraries, we recommend to run TestCov from its repository.
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88

## Usage

To check that TestCov is working as expected, run from the repository root:

```bash
bin/testcov --no-isolation --test-suite "test/suites/suite-simple-if.zip" "test/test_simple-if.c"
```

This should output the following:
```
⏳ Executing tests...
✔️  Done!

---Results---
Tests run: 2
Coverage: 100.0%
Number of goals: 2
Result: DONE
```

The output tells you:
- the number of test-cases that were executed ("Tests run: 2")
- the coverage achieved by these test executions ("Coverage: 100.0%")
- the number of test goals in the program ("Number of goals: 2")
- the result of TestCov (Result: DONE).

The above command-line uses parameter `--no-isolation` to turn of isolation of test execution.
We use this parameter to make sure that if the command fails it is some issue with your installation of TestCov,
and not some issue with BenchExec or your cgroups configuration.

If above command works, but the following command fails, it is very likely that your system is not configured
as [required by BenchExec](https://github.com/sosy-lab/benchexec/blob/master/doc/INSTALL.md).

```bash
bin/testcov --test-suite "test/suites/suite-simple-if.zip" "test/test_simple-if.c"
```

Run `bin/testcov --help` to get additional information
about configuration parameters.

89 90
## Details

91
**Test Execution.**
92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116
For test execution,
TestCov creates a test harness (in C) that reads test values from standard input.
TestCov compiles the original program with the test harness.
This allows TestCov to efficiently feed test inputs to the program under test.
Test inputs are read from a given test suite. Test suites must be specified in the
exchangable [test-format](https://gitlab.com/sosy-lab/software/test-format) and given as a single zip-file (e.g., `suite.zip`).
TestCov is agnostic about the directory structure in the test-suite zip:
It recursively searches the zip for xml files that describe individual test cases, identified through their root element.
That means, that TestCov sees both of the following as valid test suites:

```
suite-1.zip
|- metadata.xml
|- test1.xml
|- test2.xml
```

```
suite-2.zip
|- suite/
    |- metadata.xml
    |- tests/
        |- t1.xml
        |- t2.xml
```
Thomas Lemberger's avatar
Thomas Lemberger committed
117

118
**Coverage.** By default, TestCov measures branch coverage.
119 120 121
The coverage goal to measure can be define with parameter `--goal FILE`.
Example coverage-goal definitions can be found in [contrib/goal_files](contrib/goal_files).

122
**Results.**
123 124 125 126 127 128
Upon completion,
TestCov reports the test coverage achieved by the executed test suite
and whether a test covered a call to an error function (currently, `__VERIFIER_error`).
In addition, file `output/results.json` gives detailed information about each executed test
(runtime of that test, individual coverage achieved by that test, etc.)
and a reduced test suite is produced at `output/reduced-suite.zip`.
Thomas Lemberger's avatar
Thomas Lemberger committed
129

130

Thomas Lemberger's avatar
Thomas Lemberger committed
131 132 133 134
## Support

If you find something not working or know of some improvements,
we're always happy about new issues or pull requests!