Coverage reports inacurate with multiple runs of `tox`

Summary

When running tox without specifying an environment list, and tox runs pytest in more than one python environment, we only collect the coverage report from the last pytest invocation.

Steps to reproduce

• Install multiple versions of python
• Run tox in your BuildStream checkout

What is the current bug behavior?

Coverage outputs a report to a .coverage file, if the .coverage file exists then it is overwritten. Effectively we only end up collecting the report from the last run.

What is the expected correct behavior?

We extract all relevant coverage.

Ideally after running tox, there is one .coverage file with combined coverage from all of the runs, this way the .coverage file feeds perfectly into the .gitlab-ci.yml mechanics which further combine that report with reports on other machines.

Or, we can change the .gitlab-ci.yml to accommodate a scenario where running tox produces multiple coverage files.

Possible fixes

Here are some fixes and notes on them:

• Use an envlist trick with additional environments

  I've spotted an example where someone setup a "testenv:start" and "testenv:end" to cleanup and record state between runs and then combine at the end.

  This is undesirable because we want to be able to invoke specific test environments like tox -e py36, tox -e py36,lint or tox -e docs

• Set the parallel option to true

  This is advertised to output a process specific report, however it seems this has no effect since we already set parallelism = multiprocessing which seems to automate the per process report collation already.

• Enable the append option

  I cant find a configuration file option for this but there is a CLI option which says that an existing .coverage file is not wiped but instead appended to.

  This would be dangerous if we succeed to parallelize environment runs, not sure if coverage expects this scenario.

@BenjaminSchubert suggests that we give each environment a separate directory to write to.

---