Add performance regression testsuite

Description

The performance regression testsuite is thought to run different tests (at the moment only unit tests) for a lot of different input files to test the performance of the code for various parts of the parameter space.

It can be called as make check-performance from the build dir, or also using the oct-run_performance_testsuite.sh script from the installed bin directory (no parameters needed).

Some parameters can be used to tweak the behavior (when setting them as environment variables):

• TESTS: by default all, but can also be a list of tests to run
• SLURM: if set to true, salloc + srun is used to run the performance regression testsuite through slurm
• SALLOC: if set to false, do not use salloc for running make
• NODES: number of nodes to run on if using slurm
• TASKS: how many tasks are available to run in parallel - useful with and without slurm
• NPROCS: number of processors to use for each test run -> number of jobs to run in parallel is then given bi TASKS/NPROCS

Example: make check-performance SLURM=true NODES=2 TASKS=64

If running each test in parallel, one can use make check-performance SLURM=true NODES=2 TASKS=64 NPROCS=2.

When using the performance regression testsuite on slurm system inside a sbatch job, one can use TESTS=batch_ops SLURM=true SALLOC=false NODES=1 TASKS=40 NPROCS=1 $OCTOPUS_HOME/bin/oct-run_performance_testsuite.sh (in this example, only the batch_ops tests are run).

Also, parameters of the tests can be overridden by specifying OPRT_parameter_name environment variables with valid YAML code defining the corresponding parameter value array as in the combinations.yaml files.

News snippet

Add performance regression testsuite

Checklist

• I have checked that my code follows the Octopus coding standards
• I have added tests for all the new features added in this request.

testsuite/performance/oct-run_performance_testsuite.sh.in

@@ -56,7 +56,8 @@ SLURM=${SLURM:-false}
 SALLOC=${SALLOC:-true}
 TESTS=${TESTS:-all}
 NODES=${NODES:-1}
-JOBS=${JOBS:-1}
+TASKS=${TASKS:-1}
+NPROCS=${NPROCS:-1}
 comparison="no"
 
 while getopts "hlscp:d:t:" opt ; do
@@ -110,8 +111,8 @@ echo "*****************************************"
 
 if [[ "$SLURM" == "true" && "$SALLOC" == "true" ]]; then
   cd $run_directory
-  salloc --nodes $NODES --ntasks-per-node $(($JOBS/$NODES)) -K15 make -f $testsuite/Makefile.performance -j $JOBS TESTS=$TESTS SLURM=$SLURM bin_directory=$bin_directory bin_testsuite=$bin_testsuite testsuite_dir=$testsuite
+  salloc --nodes $NODES --ntasks-per-node $(($TASKS/$NODES)) -K15 make -f $testsuite/Makefile.performance -j $(($TASKS/$NPROCS)) bin_directory=$bin_directory bin_testsuite=$bin_testsuite testsuite_dir=$testsuite
 else
   cd $run_directory
-  make -f $testsuite/Makefile.performance -j $JOBS TESTS=$TESTS SLURM=$SLURM bin_directory=$bin_directory bin_testsuite=$bin_testsuite testsuite_dir=$testsuite
+  make -f $testsuite/Makefile.performance -j $(($TASKS/$NPROCS)) bin_directory=$bin_directory bin_testsuite=$bin_testsuite testsuite_dir=$testsuite
 fi