Skip to content

GitLab

Last edited by Ye Luo
Page history

Test suite and test farm

Quantum ESPRESSO has an automatic test suite based around testcode, a python script originally developed by J. Spencer [ GitHub repository]. The test suite is used to ensure continuous integration and long-term numerical stability of the code. The test suite is also used by several buildbots, running on a machine test farm, to automatically test the code nightly after a commit to the repository has been detected.

The test suite can be found in q-e/test-suite. To run the test suite, cd to that directory and type make. You will be given multiple options. For example if you want to run all the EPW tests in parallel, type:

make run-tests-epw-parallel

How to add tests for a new executable

Let us take the example of adding a new test for the TDDFPT module. In the following, PROG=tddfpt

  • extract-PROG.x: This script extracts the physical quantities from the output and parse it in a format for the testcode.py script. The script need to contain all the different output you want to parse (for chain calculations). For example, in this case we want to parse the output of pw.x, turbo_lanczos.x and turbo_spectrum.x. It is crucial to add as many parameter to be tested as possible to increase the coverage of code features.
  • run-PROG.sh: This bash script contains the paths of the different programs and source the ENVIRONMENT file
  • jobconfig: You need to edit this file to add all the new tests as well as the new program. You can chain different programs with different output in one test. In this case we added
        [tddfpt_*/]
        program = TDDFPT

This means that all the new tests related to TDDFPT must be placed in a folder with a name that starts with tddfpt_. You can also add it to a new category.

  • userconfig.tmp: This file contains the accepted accuracies for the different physical quantities defined in extract-PROG.x. You need to add a new section for your program. For the tolerance variable, the first column is the absolute accepted value, the second one is the relative accepted value and the third column contains the name of the physical quantity as defined earlier. Note that you need to add the values for all the code that you intend to test. In our case we need to add variable from pw.x as well (although already defined for other program). To estimate the acceptable tolerance, it is advised to start with very strict tolerance (very low value, e.g. 1d-6 or so) and then make some local tests (for example comparing the results in sequential or in parallel). One can then raise slightly the accepted tolerance.
  • Makefile: One need to add a line to check for potential pseudopotential to be downloaded by adding the line @./check_pseudo.sh tddfpt_.
  • PROG_TEST: Create one folder for each new test you want to add following the convention "PROG_TEST" for names. In our case we create a folder name tddfpt_CH4. The folder must contain all the input files and the reference files. Needed pseudopotential files must be available either in the "pseudo/" local directory or in www.quantum-espresso.org (a script takes care of downloading them): please do not add PP files to the repository unless really needed. The reference files must have a name that starts with benchmark.out.git.inp=. However, the easiest is to run the test suite for that test and the code will tell you what is the name he expects to have. You can then rename your reference output with that name. In our case we will therefore do
        make run-custom-test testdir=tddfpt_CH4

We can then rename the output by doing

        cp test.out.030117.inp=CH4.pw-in.args=1 benchmark.out.git.inp=CH4.pw-in.args=1

We now have a reference file for the first step of the calculation. We can do the same for the two other steps.

Once this is done, we can clean all the unwanted files and are left with a clean folder that can be committed to the git repository. In our case the test folder contains the following files:

        benchmark.out.git.inp=CH4.pw-in.args=1      
        benchmark.out.git.inp=CH4.tddfpt_pp-in.args=3 
        benchmark.out.git.inp=CH4.tddfpt-in.args=2 
        CH4.tddfpt-in    
        CH4.pw-in                                      
        CH4.tddfpt_pp-in

It is very important to then re-run the tests in parallel (4 cores) to be sure that the results are within the accepted tolerance.

How to add tests for an existing executable

You have to create a new folder following the convention PROG and TEST and then follow the structure outline above. If you want to test new physical quantities, you need to parse them using the script extract-PROG.x. Finally, the new test should be added to jobconfig (not always needed).

Test farm

As of Aug. 2020, Quantum ESPRESSO has a new nightly automatic test farm running here. If new commits are detected on the develop branch of GitLab during the day, the nightly test-farm is activated at 1 a.m.

Currently (Feb.2021) tested configurations:

  • GCC v.10.2, serial
  • GCC v.10.2, MPI (openmpi v.4.0.4)
  • GCC v.10.2, MPI (openmpi v.4.0.4), with HDF5
  • GCC v.10.2, MPI and OpenMP
  • GCC v.7.5, MPI (openmpi v.3.1.3), with debug flags on
  • Intel ifort v.19, with MKL, MPICH v.2.3
  • Intel ifort v.19, with MKL, Intel MPI
  • PGI v.19.10, MPI (openmpi v3.1.3)
  • PGI v.19.10, MPI (openmpi v3.1.3), compiled for CUDA
  • PGI v.20.11, MPI (openmpi v3.1.3), compiled for CUDA

CTest (experimental)

When CMake is used to build QE, unit and system tests can be enabled by -DQE_ENABLE_TEST=ON (default is ON). These tests can be run via ctest in the build directory after building all the executables.

ctest -j16 --output-on-failure -L unit # to run unit tests only

System tests run actual simulations from test-suite and use the same validation scheme.

ctest -j16 --output-on-failure -L "pw|cp" -LE epw # to run pw and cp system tests only
  • Without -jN, all the tests run sequentially on 1 CPU slot. With -jN, tests runs concurrently on N CPU slots with minimal over-subscription based on the number of MPI ranks and OpenMP threads.
  • -L <labels> selects the test labels, -LE <labels> excluses lables.
  • -R <pattern> selects tests by name, -E <pattern> filters tests by name
  • --output-on-failure brings the output of failed tests to the terminal. Testing/Temporary/LastTest.log stores all the test output from the last ctest run.
  • On nodes with more than 1 GPU, using --resource-spec-file ../test-suite/gpu-resource-example.json option to distribute tests. The json file can be customized based on the node.
  • test labels of both coarse and fine categories are added. For example, pw label runs all the tests under pw_* and pw_atom label runs tests only under pw_atom
  • Optional CMake variables ESPRESSO_PSEUDO, NETWORK_PSEUDO, TESTCODE_NPROCS(default 4) and TESTCODE_NTHREADS(default 1) to override default values.
  • Environment variable CTEST_XX_OPTIONS may be used to pass additional command line options. For example, CTEST_PW_OPTIONS="-ndiag 1" passes -ndiag 1 to pw.x when it is launched by ctest.