Tools for calculating test coverage and integration in CI

Related: #11 (closed).

This MR implements coverage measuring for the Albert test suite using bisect_ppx.

It contains several things:

In the first commit:

• Adds dependency to bisect_ppx in the opam file with the { with-test } qualifier.

• A series of make targets, similar to those forthcoming for the Tezos node (tezos/tezos!1404 (merged))

  • make coverage-setup:
    1. creates the coverage output dir _coverage_output if it does not exit
    2. instruments the one dune file of Albert (extraction/dune)
    3. prods the user to setup the BISECT_FILE environment variable
  • the user can now compile Albert (using make) and have a compiler instrumented for coverage
  • the user can now run some tests, using e.g. make test and coverage data is generated
  • at this point, an html report can be generated using make coverage-report-html, or on stdout with make coverage-report
  • make coverage-clean that which removes coverage files and html reports. I've added this target as a dependency to clean.
  • Additionally, I've also modified examples/Makefile to add a new target test, that recompiles the test files even if the corresponding .tz file is present. The root Makefile calls this target when make test is called.

• Two scripts:

  • scripts/instrument_dune_bisect.sh: stolen and modified from the Tezos MR referenced above. Called with no arguments, it adds the bisect_ppx stanza to extraction/dune. With arguments, it instruments the given dune files.
  • scripts/test_coverage_report.sh: this command automates the process for generating a coverage for the full test suite. it clean previous coverage logs/reports, setups coverage, recompiles the compiler if necessary (if previous compilation did not have the instrumentation e.g.), runs the test suite, and makes an html and ascii report.

In the second commit, I use the above to get coverage information for the test suite over all branches in the CI: the HTML report as an artifact of the job, the ascii output in the job log and I've also used the coverage configuration of gitlab, which gives coverage information in the GitLab UI:

• I've changed the .gitlab-ci.yml so that coverage scripts/test_coverage_report.sh script is run for the coq job (i.e. I only generate coverage information when building using the latest coq version)
  • maybe it would be nicer to calculate coverage in a separate job, but I opted against it for simplicity, and as the tests are already run in the build targets (jobs coq, and coq:* due to the --with-test flag)
  • to make the test_coverage_report.sh run in the working directory of the job, I added the flag --inplace-build to the opam pin of Albert : i'm not familiar enough with opam and the gitlab ci to know if this gives any unintended consequences!
• The generated HTML report should be available as an artifact for 3 weeks
• The ascii output is available in the job log
• The coverage stanza of the .gitlab-ci.yml exposes coverage information in the GitLab interface and gives access to cute little coverage badges. For instance:
  • For this branch: ![coverage report](https://gitlab.com/nomadic-labs/albert/badges/arvid@coverage/coverage.svg) gives
  • For branch master: ![coverage report](https://gitlab.com/nomadic-labs/albert/badges/master/coverage.svg) gives
  • On the right hand side of the job log and on the MR page (see coverage below in the associated pipeline)
  • It seems that you also get some kind of diff of coverage, however I'm not sure what baseline is used (the coverage information of the targeted branch presumably)

Limitations

One annoying thing is that .ml files extracted from the Coq std library (e.g. BinInt.ml) is included in the report and pulls down our coverage score :> Possible solutions:

1. deleting those .ml files before generating the report (not sure if this excludes them from the report though) -> does not work
2. adding [@@@coverage exclude_file] to the top of those files (http://aantron.github.io/bisect_ppx/#Exclusion). However, this would have to be done after extraction but before ml compilation, so I would have to junk around in the Makefile.
3. either solution would require specifying (manually?) what files are from albert and which are from the coq std library.