Make available ctest test performance to gitlab.com (junit.xml). Fail if single test is too long.
Description
Two things happen in this MR.
1. Ask ctest to report test performance in a standard junit file format, and upload that file to Gitlab.
Gitlab understands that file format, and provides new ways of browsing test performance, for example at https://gitlab.com/octopus-code/octopus/-/pipelines/1364734652/test_report:
One can drill into a pipeline, such as https://gitlab.com/octopus-code/octopus/-/pipelines/1364734652/test_report?job_name=foss_cmake%3A+%5Bfoss2022a-serial%2C+foss-full%5D and gets to see the tests, ordered according to CPU time use:
It remains to be seen if --- in case a test fails --- more information about that test becomes available.
2. List slowest jobs in CI output, and fail if too slow
We have added a short Python program that lists the 5 slowest tests in the test output.
5 longest tests:
name time (s)
---------------------------------------
periodic_systems/07-mgga 270
periodic_systems/06-h2o_pol_lr 221
periodic_systems/27-Ar 199
optimal_control/03-localtarget 189
photo_electron/01-h1d_lin 187
This script will also create a fail if the slowest test takes longer than 1000 seconds. (1000 seconds and 5 tests are configurable).
Discussion
I believe point 1. should definitely be merged: we need to explore what tools we get from gitlab for free, for that we need to record some junit data - in particular for failing tests, so it would be good to have this in main asap.
Point 2 is a nice prototype for the feature request #1037 (closed) . I would also suggest to merged it, but we could remove it (or put it in a separate merge request if desired).
There is more discussion in the comments below.
Addresses #1044 (closed)
Closes #1044 (closed)
Relates to #1037 (closed) .