Add performance regression testsuite

Description

The performance regression testsuite is thought to run different tests (at the moment only unit tests) for a lot of different input files to test the performance of the code for various parts of the parameter space.

It can be called as make check-performance from the build dir, or also using the oct-run_performance_testsuite.sh script from the installed bin directory (no parameters needed).

Some parameters can be used to tweak the behavior (when setting them as environment variables):

• TESTS: by default all, but can also be a list of tests to run
• SLURM: if set to true, salloc + srun is used to run the performance regression testsuite through slurm
• SALLOC: if set to false, do not use salloc for running make
• NODES: number of nodes to run on if using slurm
• TASKS: how many tasks are available to run in parallel - useful with and without slurm
• NPROCS: number of processors to use for each test run -> number of jobs to run in parallel is then given bi TASKS/NPROCS

Example: make check-performance SLURM=true NODES=2 TASKS=64

If running each test in parallel, one can use make check-performance SLURM=true NODES=2 TASKS=64 NPROCS=2.

When using the performance regression testsuite on slurm system inside a sbatch job, one can use TESTS=batch_ops SLURM=true SALLOC=false NODES=1 TASKS=40 NPROCS=1 $OCTOPUS_HOME/bin/oct-run_performance_testsuite.sh (in this example, only the batch_ops tests are run).

Also, parameters of the tests can be overridden by specifying OPRT_parameter_name environment variables with valid YAML code defining the corresponding parameter value array as in the combinations.yaml files.

News snippet

Add performance regression testsuite

Checklist

• I have checked that my code follows the Octopus coding standards
• I have added tests for all the new features added in this request.

performance_testsuite/Makefile.performance

@@ -11,7 +11,7 @@ else
   MPIEXEC ?= mpiexec -n $(NPROCS)
 endif
 OCTOPUS ?= ${bin_directory}/octopus
-run_directory ?= ./
+bin_testsuite ?= ./
 
 TESTS ?= all
 
@@ -23,7 +23,7 @@ all: testsuite timings.yaml combinations.yaml
 testsuite: $(targets)
 
 targets.inc:
-	testsuite=${testsuite_dir} python3 ${run_directory}/create_combinations.py $(TESTS)
+	testsuite=${testsuite_dir} python3 ${bin_testsuite}/create_combinations.py $(TESTS)
 
 %/profiling/time.yaml:
 	@mkdir -p $(@D)
@@ -42,7 +42,7 @@ combinations.yaml: timings.yaml
 
 comparison: $(REFERENCE_FILE) timings.yaml
 	@echo Comparing results to $(REFERENCE_FILE).
-	@python3 ${run_directory}/compare_results.py $(REFERENCE_FILE) timings.yaml
+	@python3 ${bin_testsuite}/compare_results.py $(REFERENCE_FILE) timings.yaml
 
 clean:
 	rm -rf runs targets.inc timings.yaml combinations.yaml