Skip to content

WIP: Update to testsuite tolerances by systematically reducing them and removal of the default tolerance

Nicolas Tancogne-Dejean requested to merge testsuite2.0 into develop

The usage of a default tolerance has lead to many tests having a much larger tolerance than the minimum one required to pass the corresponding matches in any correct build. As a consequence, many tests are not suitable for regression testing, as large changes in the values calculated by the code will not be detected.

On the other hand, several tests are not well designed, because they give very different results depending on the architecture/compiler/compilation options. They therefore have a large spread of the calculated values, which in turn requires large tolerances to pass successfully, which can hide further problems.

The first problem can be addressed by systematically reducing the tolerances of all the tests to the minimum value required for them to pass on all the builders of the test farm. This requires that one tolerance be set per match, thus making the default tolerance useless.

The second problem is more complicated to address and to solve it for a given test requires a developer to carefully analyze the results of that test. But before such tests can be fixed, they need to be identified. Since the spread is directly related to the minimum tolerance required for the test to pass, having all the spreads written explicitly in the test files makes it very simple to use a script to automatically identify all the tests that have unusually high spreads.

Merge request reports