Skip to content

Make available ctest test performance to gitlab.com (junit.xml). Fail if single test is too long.

Description

Two things happen in this MR.

1. Ask ctest to report test performance in a standard junit file format, and upload that file to Gitlab.

Gitlab understands that file format, and provides new ways of browsing test performance, for example at https://gitlab.com/octopus-code/octopus/-/pipelines/1364734652/test_report:

Screenshot_2024-07-10_at_11.46.15

One can drill into a pipeline, such as https://gitlab.com/octopus-code/octopus/-/pipelines/1364734652/test_report?job_name=foss_cmake%3A+%5Bfoss2022a-serial%2C+foss-full%5D and gets to see the tests, ordered according to CPU time use:

Screenshot_2024-07-10_at_11.46.49

It remains to be seen if --- in case a test fails --- more information about that test becomes available.

2. List slowest jobs in CI output, and fail if too slow

We have added a short Python program that lists the 5 slowest tests in the test output.

Example output:

5 longest tests:
name                           time (s)
---------------------------------------
periodic_systems/07-mgga            270
periodic_systems/06-h2o_pol_lr      221
periodic_systems/27-Ar              199
optimal_control/03-localtarget      189
photo_electron/01-h1d_lin           187

This script will also create a fail if the slowest test takes longer than 1000 seconds. (1000 seconds and 5 tests are configurable).

Discussion

I believe point 1. should definitely be merged: we need to explore what tools we get from gitlab for free, for that we need to record some junit data - in particular for failing tests, so it would be good to have this in main asap.

Point 2 is a nice prototype for the feature request #1037 (closed) . I would also suggest to merged it, but we could remove it (or put it in a separate merge request if desired).

There is more discussion in the comments below.

Addresses #1044 (closed)

Closes #1044 (closed)

Relates to #1037 (closed) .

Edited by Hans Fangohr

Merge request reports

Loading