Quantum ESPRESSO has an automatic test suite based around
testcode, a python script originally developed by J. Spencer [ GitHub repository].
The test suite is used to ensure continuous integration and long-term numerical stability of the code. The test suite is also used by several
on a machine test farm, to automatically test the code nightly after a commit to the repository has been detected.
The test suite can be found in
q-e/test-suite. To run the test suite,
cd to that directory and type
make. You will be given multiple options.
For example if you want to run all the EPW tests in parallel, type:
How to add tests for a new executable
Let us take the example of adding a new test for the TDDFPT module. In the following, PROG=tddfpt
extract-PROG.x: This script extracts the physical quantities from the output and parse it in a format for the testcode.py script. The script need to contain all the different output you want to parse (for chain calculations). For example, in this case we want to parse the output of
turbo_spectrum.x. It is crucial to add as many parameter to be tested as possible to increase the coverage of code features.
run-PROG.sh: This bash script contains the paths of the different programs and source the
jobconfig: You need to edit this file to add all the new tests as well as the new program. You can chain different programs with different output in one test. In this case we added
[tddfpt_*/] program = TDDFPT
This means that all the new tests related to TDDFPT must be placed
in a folder with a name that starts with
tddfpt_. You can also add
it to a new category.
userconfig.tmp: This file contains the accepted accuracies for the different physical quantities defined in
extract-PROG.x. You need to add a new section for your program. For the tolerance variable, the first column is the absolute accepted value, the second one is the relative accepted value and the third column contains the name of the physical quantity as defined earlier. Note that you need to add the values for all the code that you intend to test. In our case we need to add variable from
pw.xas well (although already defined for other program). To estimate the acceptable tolerance, it is advised to start with very strict tolerance (very low value, e.g. 1d-6 or so) and then make some local tests (for example comparing the results in sequential or in parallel). One can then raise slightly the accepted tolerance.
Makefile: One need to add a line to check for potential pseudopotential to be downloaded by adding the line
PROG_TEST: Create one folder for each new test you want to add following the convention "PROG_TEST" for names. In our case we create a folder name
tddfpt_CH4. The folder must contain all the input files and the reference files. Needed pseudopotential files must be available either in the "pseudo/" local directory or in www.quantum-espresso.org (a script takes care of downloading them): please do not add PP files to the repository unless really needed. The reference files must have a name that starts with
benchmark.out.git.inp=. However, the easiest is to run the test suite for that test and the code will tell you what is the name he expects to have. You can then rename your reference output with that name. In our case we will therefore do
make run-custom-test testdir=tddfpt_CH4
We can then rename the output by doing
cp test.out.030117.inp=CH4.pw-in.args=1 benchmark.out.git.inp=CH4.pw-in.args=1
We now have a reference file for the first step of the calculation. We can do the same for the two other steps.
Once this is done, we can clean all the unwanted files and are left with a clean folder that can be committed to the git repository. In our case the test folder contains the following files:
benchmark.out.git.inp=CH4.pw-in.args=1 benchmark.out.git.inp=CH4.tddfpt_pp-in.args=3 benchmark.out.git.inp=CH4.tddfpt-in.args=2 CH4.tddfpt-in CH4.pw-in CH4.tddfpt_pp-in
It is very important to then re-run the tests in parallel (4 cores) to be sure that the results are within the accepted tolerance.
How to add tests for an existing executable
You have to create a new folder following the convention PROG and
TEST and then follow the structure outline above. If you want to
test new physical quantities, you need to parse them using the script
extract-PROG.x. Finally, the new test should be added to
jobconfig (not always needed).
As of Aug. 2020, Quantum ESPRESSO has a new nightly automatic test farm running here. If new commits are detected on the develop branch of GitLab during the day, the nightly test-farm is activated at 1 a.m.
Currently (Feb.2021) tested configurations:
- GCC v.10.2, serial
- GCC v.10.2, MPI (openmpi v.4.0.4)
- GCC v.10.2, MPI (openmpi v.4.0.4), with HDF5
- GCC v.10.2, MPI and OpenMP
- GCC v.7.5, MPI (openmpi v.3.1.3), with debug flags on
- Intel ifort v.19, with MKL, MPICH v.2.3
- Intel ifort v.19, with MKL, Intel MPI
- PGI v.19.10, MPI (openmpi v3.1.3)
- PGI v.19.10, MPI (openmpi v3.1.3), compiled for CUDA
- PGI v.20.11, MPI (openmpi v3.1.3), compiled for CUDA
When CMake is used to build QE, unit and system tests can be enabled by
-DQE_ENABLE_TEST=ON (default is ON). These tests can be run via ctest in the build directory after building all the executables.
ctest -j16 --output-on-failure -L unit # to run unit tests only
System tests run actual simulations from test-suite and use the same validation scheme.
ctest -j16 --output-on-failure -L "pw|cp" -LE epw # to run pw and cp system tests only
-jN, all the tests run sequentially on 1 CPU slot. With
-jN, tests runs concurrently on N CPU slots with minimal over-subscription based on the number of MPI ranks and OpenMP threads.
-L <labels>selects the test labels,
-LE <labels>excluses lables.
-R <pattern>selects tests by name,
-E <pattern>filters tests by name
--output-on-failurebrings the output of failed tests to the terminal.
Testing/Temporary/LastTest.logstores all the test output from the last ctest run.
- On nodes with more than 1 GPU, using
--resource-spec-file ../test-suite/gpu-resource-example.jsonoption to distribute tests. The json file can be customized based on the node.
- test labels of both coarse and fine categories are added. For example,
pwlabel runs all the tests under
pw_atomlabel runs tests only under pw_atom
- Optional CMake variables
TESTCODE_NPROCS(default 4) and
TESTCODE_NTHREADS(default 1) to override default values.
- Environment variable CTEST_XX_OPTIONS may be used to pass additional command line options. For example,
-ndiag 1to pw.x when it is launched by ctest.