Fail testsuite

Description

This update to the testsuite allows to test for failing runs. These can be either due to wrong input files, or due to missing components (i.e. SPARSKIT, etc). Deliberately failing runs are to be defined as "FailingInput". Error messages are checked by "errormatch" lines.

News snippet

This update to the testsuite allows to test for failing runs.

Checklist

• I have checked that my code follows the Octopus coding standards
• I have added tests for all the new features added in this request.

Closes #153 (closed)

changed milestone to %10.0

added Testsuite label

added 1 commit

• f1eb3ec7 - error_tests/Makefile.am added

Compare with previous version

Codecov Report

Merging #614 into develop will decrease coverage by 0.23%. The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop     #614      +/-   ##
===========================================
- Coverage    71.07%   70.83%   -0.24%     
===========================================
  Files          507      499       -8     
  Lines        98807    96242    -2565     
===========================================
- Hits         70223    68177    -2046     
+ Misses       28584    28065     -519     

Impacted Files	Coverage Δ
src/system/system_mxll.F90	0.00% <0.00%> (-76.96%)
src/states/states_mxll.F90	0.00% <0.00%> (-59.10%)
src/hamiltonian/maxwell_boundary_op.F90	0.00% <0.00%> (-49.16%)
src/system/output_mxll_inc.F90	0.37% <0.00%> (-45.87%)
src/grid/subarray_inc.F90	59.57% <0.00%> (-40.43%)
src/basic/accel_inc.F90	63.15% <0.00%> (-31.58%)
src/math/maxwell_function.F90	0.00% <0.00%> (-25.70%)
src/states/states_mxll_restart.F90	0.00% <0.00%> (-21.76%)
src/grid/boundaries_inc.F90	74.35% <0.00%> (-21.09%)
src/grid/vtk_inc.F90	0.00% <0.00%> (-20.14%)
... and 110 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update e37d40a...c87e35c. Read the comment docs.

added 1 commit

• 98342f34 - match updates

Compare with previous version

added 1 commit

• 06572d6c - fixed valgrind pfft problem

Compare with previous version

added 1 commit

• 8f2d4633 - Another Options_skip update

Compare with previous version

added 1 commit

• 243d4cc2 - typos and some comments

Compare with previous version

resolved all threads

resolved all threads

approved this merge request

@sohlmann Sebastian, did you check whether anything breaks your postprocessing of the test outputs?

No, but the YAML output isn't changed, so it should work.

The last commit looks fine in the web app: https://octopus-code.org/testsuite/commits/hash/243d4cc2afd8125e900e82199bc0ebe7341670e7

unmarked as a Work In Progress

changed the description

approved this merge request

I was now reviewing this MR and it's not clear to me what happens if there are two options in a test, but only one is present. My understanding is that the error checks will be performed in that case, even for the available option, in which case the test will fail.