Coverage reports inacurate with multiple runs of `tox`
tox without specifying an environment list, and
pytest in more than one python environment, we only collect the coverage report from the last
Steps to reproduce
- Install multiple versions of python
toxin your BuildStream checkout
What is the current bug behavior?
Coverage outputs a report to a
.coverage file, if the
.coverage file exists then it is overwritten. Effectively we only end up collecting the report from the last run.
What is the expected correct behavior?
We extract all relevant coverage.
Ideally after running
tox, there is one
.coverage file with combined coverage from all of the runs, this way the .coverage file feeds perfectly into the
.gitlab-ci.yml mechanics which further combine that report with reports on other machines.
Or, we can change the
.gitlab-ci.yml to accommodate a scenario where running
tox produces multiple coverage files.
Here are some fixes and notes on them:
Use an envlist trick with additional environments
I've spotted an example where someone setup a "testenv:start" and "testenv:end" to cleanup and record state between runs and then combine at the end.
This is undesirable because we want to be able to invoke specific test environments like
tox -e py36,
tox -e py36,lintor
tox -e docs
This is advertised to output a process specific report, however it seems this has no effect since we already set
parallelism = multiprocessingwhich seems to automate the per process report collation already.
I cant find a configuration file option for this but there is a CLI option which says that an existing
.coveragefile is not wiped but instead appended to.
This would be dangerous if we succeed to parallelize environment runs, not sure if coverage expects this scenario.
@BenjaminSchubert suggests that we give each environment a separate directory to write to.