Add performance regression testsuite

Description

The performance regression testsuite is thought to run different tests (at the moment only unit tests) for a lot of different input files to test the performance of the code for various parts of the parameter space.

It can be called as make check-performance from the build dir, or also using the oct-run_performance_testsuite.sh script from the installed bin directory (no parameters needed).

Some parameters can be used to tweak the behavior (when setting them as environment variables):

• TESTS: by default all, but can also be a list of tests to run
• SLURM: if set to true, salloc + srun is used to run the performance regression testsuite through slurm
• SALLOC: if set to false, do not use salloc for running make
• NODES: number of nodes to run on if using slurm
• TASKS: how many tasks are available to run in parallel - useful with and without slurm
• NPROCS: number of processors to use for each test run -> number of jobs to run in parallel is then given bi TASKS/NPROCS

Example: make check-performance SLURM=true NODES=2 TASKS=64

If running each test in parallel, one can use make check-performance SLURM=true NODES=2 TASKS=64 NPROCS=2.

When using the performance regression testsuite on slurm system inside a sbatch job, one can use TESTS=batch_ops SLURM=true SALLOC=false NODES=1 TASKS=40 NPROCS=1 $OCTOPUS_HOME/bin/oct-run_performance_testsuite.sh (in this example, only the batch_ops tests are run).

Also, parameters of the tests can be overridden by specifying OPRT_parameter_name environment variables with valid YAML code defining the corresponding parameter value array as in the combinations.yaml files.

News snippet

Add performance regression testsuite

Checklist

• I have checked that my code follows the Octopus coding standards
• I have added tests for all the new features added in this request.

performance_testsuite/create_combinations.py

@@ -63,6 +63,13 @@ def get_combinations(test_name, path='tests'):
     with open(os.path.join(path, filename), 'r') as f_in:
         data = yaml.safe_load(f_in)
     combinations_dict = data['parameters']
+    for parameter in combinations_dict:
+        # check environment variable to override
+        variable = 'OPRT_' + parameter
+        if variable in os.environ:
+            print("Information: {} is overridden by environment variable"
+                  .format(parameter))
+            combinations_dict[parameter] = yaml.safe_load(os.environ[variable])
     combinations_list = modify_combinations(
         outer_product_from_dict(combinations_dict), test_name)
     combinations = {}