Tools for calculating test coverage and integration in CI
Related: #11 (closed).
This MR implements coverage measuring for the Albert test suite using bisect_ppx.
It contains several things:
In the first commit:
-
Adds dependency to
bisect_ppx
in the opam file with the{ with-test }
qualifier. -
A series of make targets, similar to those forthcoming for the Tezos node (tezos/tezos!1404 (merged))
-
make coverage-setup
:- creates the coverage output dir
_coverage_output
if it does not exit - instruments the one dune file of Albert (
extraction/dune
) - prods the user to setup the
BISECT_FILE
environment variable
- creates the coverage output dir
- the user can now compile Albert (using
make
) and have a compiler instrumented for coverage - the user can now run some tests, using e.g.
make test
and coverage data is generated - at this point, an html report can be generated using
make coverage-report-html
, or on stdout withmake coverage-report
-
make coverage-clean
that which removes coverage files and html reports. I've added this target as a dependency toclean
. - Additionally, I've also modified
examples/Makefile
to add a new targettest
, that recompiles the test files even if the corresponding.tz
file is present. The root Makefile calls this target whenmake test
is called.
-
-
Two scripts:
-
scripts/instrument_dune_bisect.sh
: stolen and modified from the Tezos MR referenced above. Called with no arguments, it adds the bisect_ppx stanza toextraction/dune
. With arguments, it instruments the given dune files. -
scripts/test_coverage_report.sh
: this command automates the process for generating a coverage for the full test suite. it clean previous coverage logs/reports, setups coverage, recompiles the compiler if necessary (if previous compilation did not have the instrumentation e.g.), runs the test suite, and makes an html and ascii report.
-
In the second commit, I use the above to get coverage
information for the test suite over all branches in the CI: the HTML report as an
artifact of the job, the ascii output in the job log and I've also
used the coverage
configuration of gitlab, which gives coverage
information in the GitLab UI:
- I've changed the
.gitlab-ci.yml
so that coveragescripts/test_coverage_report.sh
script is run for thecoq
job (i.e. I only generate coverage information when building using the latest coq version)- maybe it would be nicer to calculate coverage in a separate job,
but I opted against it for simplicity, and as the tests are
already run in the build targets (jobs
coq
, andcoq:*
due to the--with-test
flag) - to make the
test_coverage_report.sh
run in the working directory of the job, I added the flag--inplace-build
to theopam pin
of Albert : i'm not familiar enough with opam and the gitlab ci to know if this gives any unintended consequences!
- maybe it would be nicer to calculate coverage in a separate job,
but I opted against it for simplicity, and as the tests are
already run in the build targets (jobs
- The generated HTML report should be available as an artifact for 3 weeks
- The ascii output is available in the job log
- The
coverage
stanza of the.gitlab-ci.yml
exposes coverage information in the GitLab interface and gives access to cute little coverage badges. For instance:- For this branch:
![coverage report](https://gitlab.com/nomadic-labs/albert/badges/arvid@coverage/coverage.svg)
gives - For branch master:
![coverage report](https://gitlab.com/nomadic-labs/albert/badges/master/coverage.svg)
gives - On the right hand side of the job log and on the MR page (see coverage below in the associated pipeline)
- It seems that you also get some kind of diff of coverage, however I'm not sure what baseline is used (the coverage information of the targeted branch presumably)
- For this branch:
Limitations
One annoying thing is that .ml
files extracted from the Coq std library (e.g. BinInt.ml
) is included in the report and pulls down our coverage score :> Possible solutions:
-
deleting those-> does not work.ml
files before generating the report (not sure if this excludes them from the report though) - adding
[@@@coverage exclude_file]
to the top of those files (http://aantron.github.io/bisect_ppx/#Exclusion). However, this would have to be done after extraction but before ml compilation, so I would have to junk around in the Makefile. - either solution would require specifying (manually?) what files are from albert and which are from the coq std library.