Skip to content

Add performance regression testsuite

Sebastian Ohlmann requested to merge performance_testsuite into develop

Description

The performance regression testsuite is thought to run different tests (at the moment only unit tests) for a lot of different input files to test the performance of the code for various parts of the parameter space.

It can be called as make check-performance from the build dir, or also using the oct-run_performance_testsuite.sh script from the installed bin directory (no parameters needed).

Some parameters can be used to tweak the behavior (when setting them as environment variables):

  • TESTS: by default all, but can also be a list of tests to run
  • SLURM: if set to true, salloc + srun is used to run the performance regression testsuite through slurm
  • SALLOC: if set to false, do not use salloc for running make
  • NODES: number of nodes to run on if using slurm
  • TASKS: how many tasks are available to run in parallel - useful with and without slurm
  • NPROCS: number of processors to use for each test run -> number of jobs to run in parallel is then given bi TASKS/NPROCS

Example: make check-performance SLURM=true NODES=2 TASKS=64

If running each test in parallel, one can use make check-performance SLURM=true NODES=2 TASKS=64 NPROCS=2.

When using the performance regression testsuite on slurm system inside a sbatch job, one can use TESTS=batch_ops SLURM=true SALLOC=false NODES=1 TASKS=40 NPROCS=1 $OCTOPUS_HOME/bin/oct-run_performance_testsuite.sh (in this example, only the batch_ops tests are run).

Also, parameters of the tests can be overridden by specifying OPRT_parameter_name environment variables with valid YAML code defining the corresponding parameter value array as in the combinations.yaml files.

News snippet

Add performance regression testsuite

Checklist

  • I have checked that my code follows the Octopus coding standards
  • I have added tests for all the new features added in this request.
Edited by Martin Lueders

Merge request reports