Code coverage on the CI

This MR enables the computation of the code coverage metrics from the CI, and update the documentation.

It adds two jobs to .gitlab-ci.yml

1. Run tests sequentially (alcotest, python), generate the report and exports it as an artefact
2. Publish the report on github pages at URL https://nomadic-labs.gitlab.io/tezos/PIPELINE_NUMBER/

These new jobs can be triggered manually from the gitlab interface.

As an example:

• pipeline https://gitlab.com/nomadic-labs/tezos/pipelines/149862216/
• code coverage https://nomadic-labs.gitlab.io/tezos/149862216

Discussion

The CI script runs a separate job for each tests. The main advantage is that it's faster, and gitlab gives a synthetic interface, with the ability to relaunch test. The drawback is that the CI script is somewhat complicated (it's automatically generated from the test lists), and we don't have the ability to launch the tests locally in the same way.

It's also a bit problematic to compute the coverage. These are the three solutions I could think of.

1. Compute a coverage artefact for each test, and generate a report at each run of the CI
   • Pros: we have a report at each run
   • Cons: (much?) slower that regular run of the test, we test the instrumented code instead of the "real", makes the CI script more complex
2. Solution described above
   • Pros: simple, only add things to the CI, no need to touch the scripts for generation, can be run with gitlab-runner
   • Cons: don't run coverage everytime, slower because tests are run sequentially, need to make sure make test runs the same test suite as the CI (but it should be the case anyway)
3. A mix of 1 and 2, run tests on the CI as separate jobs, but only instrument the code for coverage branches.
   • Pros: faster
   • Cons: complex

This MR implements 2 which I think is the best compromise

Running the CI locally

Because the coverage is computed in a single job (which launches the whole test suite), it's possible to run it locally.

time gitlab-runner exec docker  test_coverage
...

...

...
Report should be available in _coverage_report/index.html
Job succeeded
gitlab-runner exec docker coverage  0.96s user 0.59s system 0% cpu 15:03.40 total

This is about 3 times faster than the CI running the same job. Unfortunately, I'm still not able to retrieve the generated artefact _coverage_output (maybe possible, I haven't looked too much yet).

Note that while this wasn't the initial intent, this new CI jobs has the side effect of allowing to launch the full test suite locally. It seems to be even much faster than running the tests on the CI as we normally do (as separate jobs).

For reference, we have roughly:

• Regular CI: 30 min
• Coverage job locally: 15mn
• Coverage job CI: 20min

So running the full test suite locally, using sequential tests and instrumented code is still twice as fast as running it on gitlab with parallel tests and non-instrumented code.

Known issues

These aren't critical for this MR and are left for future work

• Coverage isn't computed properly for flextesa tests. I think this is because processes are "sigkilled"
• lib_crypto is currently excluded (doesn't compile when instrumented)
• Computing the coverage may fail randomly when processes died abruptly and leave bogus coverage files

$ make coverage-report
4409 Info: found coverage files in '_coverage_output/'
4410  *** invalid file: '_coverage_output/819770417.coverage' error: "unexpected end of file while reading magic number"

Possible improvement

• It's annoying to trigger both the report and the publishing manually (ideally one click would be enough). Also the coverage URL is buried in the pages log, would be better if it were more visible.
• The use of gitlab cache between two executions of the pages job seems a bit ad-hoc. There is the question of how much can this cache grow. We should keep an eye on it. We could remove pages altogether as artefact browsing is enough. The reviewers suggested to keep it, we'll see if it's useful or not.
• should be updated when !1710 (merged) is completed (improve test directives in makefile)
• prettify report of make test, issue 71, 72 (orthogonal but would be useful here)
• We could think of using coveralls at some point, but it's not well-integrated with bisect_ppx on gitlab yet