Add performance regression testsuite

Description

The performance regression testsuite is thought to run different tests (at the moment only unit tests) for a lot of different input files to test the performance of the code for various parts of the parameter space.

It can be called as make check-performance from the build dir, or also using the oct-run_performance_testsuite.sh script from the installed bin directory (no parameters needed).

Some parameters can be used to tweak the behavior (when setting them as environment variables):

• TESTS: by default all, but can also be a list of tests to run
• SLURM: if set to true, salloc + srun is used to run the performance regression testsuite through slurm
• SALLOC: if set to false, do not use salloc for running make
• NODES: number of nodes to run on if using slurm
• TASKS: how many tasks are available to run in parallel - useful with and without slurm
• NPROCS: number of processors to use for each test run -> number of jobs to run in parallel is then given bi TASKS/NPROCS

Example: make check-performance SLURM=true NODES=2 TASKS=64

If running each test in parallel, one can use make check-performance SLURM=true NODES=2 TASKS=64 NPROCS=2.

When using the performance regression testsuite on slurm system inside a sbatch job, one can use TESTS=batch_ops SLURM=true SALLOC=false NODES=1 TASKS=40 NPROCS=1 $OCTOPUS_HOME/bin/oct-run_performance_testsuite.sh (in this example, only the batch_ops tests are run).

Also, parameters of the tests can be overridden by specifying OPRT_parameter_name environment variables with valid YAML code defining the corresponding parameter value array as in the combinations.yaml files.

News snippet

Add performance regression testsuite

Checklist

• I have checked that my code follows the Octopus coding standards
• I have added tests for all the new features added in this request.

performance_testsuite/compare_results.py

@@ -12,21 +12,29 @@ with open(sys.argv[1]) as reference_file:
 with open(sys.argv[2]) as current_file:
     current_data = list(yaml.safe_load_all(current_file))
 
-times = {}
+time_data = {}
 for reference_test, current_test in zip(reference_data, current_data):
     index = reference_test['schema'].index('total_time')
-    reference_time = reference_test['data']['COMPLETE_RUN'][index]
-    current_time = current_test['data']['COMPLETE_RUN'][index]
-    times[reference_test['run']] = {'reference': reference_time,
-                                    'current': current_time}
+    time_data[reference_test['run']] = {}
+    for key in reference_test['data'].keys():
+        reference_time = reference_test['data'][key][index]
+        current_time = current_test['data'][key][index]
+        time_data[reference_test['run']][key] = {
+                'reference': reference_time, 'current': current_time}
 
-any_test_slower = False
-for run, time in times.items():
-    if time['current'] < time['reference']:
-        print("Run {} is slower than reference ({} vs {}).".format(
-            run, time['current'], time['reference']))
-        any_test_slower = True
+num_slower_tests = 0
+num_tests = 0
+for run, times in time_data.items():
+    for key, time in times.items():
+        num_tests += 1
+        if time['current'] < time['reference']:
+            print("Run {}, key {} is slower than reference "
+                  "(by {:.1%}, {} s vs {} s).".format(
+                      run, key, 1-time['current']/time['reference'],
+                      time['current'], time['reference']))
+            num_slower_tests += 1
 
-if any_test_slower:
-    print("Error: some runs are slower than the reference.")
+if num_slower_tests > 0:
+    print("Error: {:d} runs of {:d} are slower than the reference.".format(
+        num_slower_tests, num_tests))
     sys.exit(1)