Add performance regression testsuite

Description

The performance regression testsuite is thought to run different tests (at the moment only unit tests) for a lot of different input files to test the performance of the code for various parts of the parameter space.

It can be called as make check-performance from the build dir, or also using the oct-run_performance_testsuite.sh script from the installed bin directory (no parameters needed).

Some parameters can be used to tweak the behavior (when setting them as environment variables):

• TESTS: by default all, but can also be a list of tests to run
• SLURM: if set to true, salloc + srun is used to run the performance regression testsuite through slurm
• SALLOC: if set to false, do not use salloc for running make
• NODES: number of nodes to run on if using slurm
• TASKS: how many tasks are available to run in parallel - useful with and without slurm
• NPROCS: number of processors to use for each test run -> number of jobs to run in parallel is then given bi
  $TASKS/$
  NPROCS

Example: make check-performance SLURM=true NODES=2 TASKS=64

If running each test in parallel, one can use make check-performance SLURM=true NODES=2 TASKS=64 NPROCS=2.

When using the performance regression testsuite on slurm system inside a sbatch job, one can use TESTS=batch_ops SLURM=true SALLOC=false NODES=1 TASKS=40 NPROCS=1 $OCTOPUS_HOME/bin/oct-run_performance_testsuite.sh (in this example, only the batch_ops tests are run).

Also, parameters of the tests can be overridden by specifying OPRT_parameter_name environment variables with valid YAML code defining the corresponding parameter value array as in the combinations.yaml files.

News snippet

Add performance regression testsuite

Checklist

• I have checked that my code follows the Octopus coding standards
• I have added tests for all the new features added in this request.

changed milestone to %10.0

added Build system Core Feature Optimization Testsuite labels

Codecov Report

Merging #781 into develop will decrease coverage by 5.57%. The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop     #781      +/-   ##
===========================================
- Coverage    70.52%   64.94%   -5.58%     
===========================================
  Files          489      476      -13     
  Lines        94526    86446    -8080     
===========================================
- Hits         66663    56145   -10518     
- Misses       27863    30301    +2438     

Impacted Files	Coverage Δ
src/math/nfft_inc.F90	0.00% <0.00%> (-100.00%)
src/grid/subarray_inc.F90	0.00% <0.00%> (-100.00%)
src/grid/partition_transfer_inc.F90	0.00% <0.00%> (-100.00%)
src/states/states_elec_parallel_inc.F90	0.00% <0.00%> (-100.00%)
src/grid/subarray.F90	0.00% <0.00%> (-97.92%)
src/basic/blacs_proc_grid.F90	0.00% <0.00%> (-95.90%)
src/poisson/poisson_libisf.F90	0.00% <0.00%> (-95.78%)
src/grid/par_vec.F90	0.00% <0.00%> (-95.73%)
src/basic/alloc_cache_low.cc	2.17% <0.00%> (-95.66%)
src/basic/accel_inc.F90	0.00% <0.00%> (-94.74%)
... and 394 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 629e79a...201da25. Read the comment docs.

Codecov Report

Merging #781 into develop will decrease coverage by 0.20%. The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop     #781      +/-   ##
===========================================
- Coverage    70.71%   70.51%   -0.21%     
===========================================
  Files          501      501              
  Lines        95672    95669       -3     
===========================================
- Hits         67656    67458     -198     
- Misses       28016    28211     +195     

Impacted Files	Coverage Δ
src/grid/subarray_inc.F90	57.77% <0.00%> (-42.23%)
src/basic/accel_inc.F90	63.15% <0.00%> (-31.58%)
src/grid/boundaries_inc.F90	78.34% <0.00%> (-16.57%)
src/grid/boundaries.F90	87.23% <0.00%> (-8.52%)
src/grid/nl_operator.F90	65.97% <0.00%> (-5.68%)
src/system/subspace_inc.F90	89.55% <0.00%> (-4.98%)
src/grid/nl_operator_inc.F90	65.21% <0.00%> (-4.79%)
src/utils/casida_spectrum.F90	74.07% <0.00%> (-3.71%)
src/states/states_elec_calc_inc.F90	73.99% <0.00%> (-1.78%)
src/basic/cuda_low.cc	91.70% <0.00%> (-1.56%)
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 9ac9c77...faede91. Read the comment docs.

changed the description

changed the description

added 2 commits

• e1be56af - Integrate comparing performance results
• 478040ba - Add option to run without salloc

Compare with previous version

added 1 commit

• 50c26c95 - Allow overriding parameters of the performance testsuite

Compare with previous version

changed the description

added 5 commits

• b2e21fbc - Integrate comparing performance results
• 66b1ad8d - Add option to run without salloc
• e5c0be31 - Allow overriding parameters of the performance testsuite
• d92e9e52 - Adapt some path variable names
• 01a672e7 - Use soft_coulomb species instead of jellium

Compare with previous version

added 1 commit

• 57134256 - Reduce number of orthogonalization tests

Compare with previous version

added 1 commit

• d61c3650 - Tweak slurm parameters

Compare with previous version

added 1 commit

• ae772b07 - Use ntasks-per-node for salloc call

Compare with previous version

added 198 commits

• ae772b07...a3d54419 - 188 commits from branch develop
• f73ff5da - Add performance regression testsuite
• 2073b23d - Integrate comparing performance results
• 498c0604 - Add option to run without salloc
• 1fe522bb - Allow overriding parameters of the performance testsuite
• b0b81a4a - Adapt some path variable names
• f52c8847 - Use soft_coulomb species instead of jellium
• 7f90a2cc - Reduce number of orthogonalization tests
• e4d20e52 - Tweak slurm parameters
• aa3bedec - Use ntasks-per-node for salloc call
• 6b360ec4 - Increase threshold for soft coulomb species

Compare with previous version

@sohlmann I'm now testing the latest changes to see if the tests now take a reasonable amount of time.

In the meantime, if it's okay with you, I will move the performance testsuite to be inside the testsuite folder, as I don't want it to be too obvious for users that there's a test for performance.

Sure, no problem. You will have to adapt the Makefiles, however.

@sohlmann Looks like this is still not behaving as expected. The serial run seems fine, with 32 jobs per node, but for the parallel run there are only 8 octopus processes on each node. Any idea?

added 1 commit

• 050384e0 - Move performance testsuite to testsuite folder.

Compare with previous version

changed the description

added 3 commits

• 3855c3c0 - Add error message if tests are not found
• 01f5ee31 - Fix path for installing tests
• fafb4f75 - Fix logic of parallel runs

Compare with previous version

added 83 commits

• fafb4f75...9ac9c773 - 69 commits from branch develop
• 284daa6a - Add performance regression testsuite
• ec851315 - Integrate comparing performance results
• 7b495e26 - Add option to run without salloc
• 08fb419f - Allow overriding parameters of the performance testsuite
• fc494985 - Adapt some path variable names
• 58395077 - Use soft_coulomb species instead of jellium
• c25cec31 - Reduce number of orthogonalization tests
• 5b634374 - Tweak slurm parameters
• 0fc62125 - Use ntasks-per-node for salloc call
• 033067c3 - Increase threshold for soft coulomb species
• 5862dee2 - Move performance testsuite to testsuite folder.
• 1651ab7d - Add error message if tests are not found
• 2a40679c - Fix path for installing tests
• faede914 - Fix logic of parallel runs

Compare with previous version

approved this merge request

resolved all threads

enabled an automatic merge when the pipeline for faede914 succeeds

mentioned in commit 6a031416

merged

changed the description

added in Changelog label