Investigate new unit testing frameworks

Closes #41 (closed).

Frameworks checked:

pFUnit

pFUnit builds and installs under Windows; OpenMP and MPI can be disabled. Built and tested FindPFUNIT.cmake. Framework requires Python for the test runner which adds more dependencies than desirable; cannot build tests without Python. Review further if Python becomes a required dependency; not worth requiring Python just for unit testing.

FLIBS/ftnunit

FLIBS is already a dependency; this may be simple to test as a source-only test framework. That would greatly simplify installation and construction of tests.

Integrating ftnunit is straightforward but fails to compile under coverage analysis on Linux which severely limits its usefulness. Experimental work has been documented and moved to contrib in case it's needed later.

zofu

zofu (https://github.com/acroucher/zofu) is simple to build, has no runtime dependencies, and integrates well with CMake builds. It provides better failure diagnostics than TOAST. The main compile-time dependencies are ninja and meson, the latter forcing an otherwise unnecessary dependency on a full Python 3.x installation. The only other issue is the installer not correctly placing Fortran .mod files. Beyond that, the test framework seems sensible and at least as usable as TOAST with more relevant features and less unavoidable overhead. Replacing meson.build with an equivalent CMakeLists.txt would eliminate the need for Python and ninja in environments where those tools are not readily available.

vegetables

See https://gitlab.com/everythingfunctional/vegetables Further analysis shows vegetables depends on fpm (Fortran package manager) and at least two direct dependencies sourced via fpm. Further, the functional style of the test suite bears no relationship to legacy or modern procedural or object-oriented Fortran, preventing tests from being used as examples of how the project code is used in practice. While the test code does not necessarily need to match the style of the project code, there is no obvious benefit to the functional style let alone a benefit exceeding the cost of learning and maintaining two very different development styles in the project. In short, the important elements provided by vegetables (basic assertions) are also provided by simpler frameworks but without as much build and cognitive overhead. It's not clear what problem vegetables solves that is not solved by other more lightweight, more conventional tools.

TOAST

In use; see https://github.com/thomasms/toast

TOAST would be more useful if it could be included as a source-only dependency, possibly without JSON support (currently unused). There are several issues with TOAST: it requires extensive CMake support to use, requires git and compilation to use, TOAST has no installer so artifacts must be referenced in the build tree. It does not show values of operands when tests fail which complicates diagnosis and resolution.

A separate issue might be opened for investigating the difficulty of simplifying TOAST and converting it to a source-only framework like ftnunit

Conclusion

TOAST and zofu are worth pursuing, specifically to (better) integrate with CMake and remove dependencies on git and meson (ninja should not be required if make is available). ftnunit would be worth pursuing if the coverage analysis problem were solved. pFUnit requires too much extra tooling to be worth pursuing. Similarly vegetables requires too much additional tooling at build time and seems to require tests be written in a style wildly different from the rest of the codebase; it's a poor fit culturally and technically for this project and its goals.

It may be reasonable to extend the zofu proof-of-concept test to an actual production test and confirm it works with the coverage analysis system (gcov/lcov). At that point, the zofu test scheme should be temporarily removed from the main code, and archived in contrib and this MR closed. A new MR should be then opened to modify TOAST and zofu to build and install with CMake and eliminate git and meson as dependencies. meson is more likely to be eliminated than git so for sustainability and portability, the existing TOAST unit tests may need to be migrated to zofu if TOAST can't be built without requiring git.