Skip to content

Draft: Test kats against reference data with parametrization

finesse importer requested to merge feature/ref-tests into develop

Related to !90 (closed).

Adds a test to assert equality between several .kat files and their paired .ref output files.

The goal is to introduce light-weight tests to compare a model's output against its expected values. To keep it light, only a few data points are to be used which can be done by using sweep passing only a select number of values rather than a range. This is in contrast to the more thorough finesse/test repository which seeks to compare full data sets.

The test file is based off of Aaron's test/reference/test3.py and uses pytest's parametrize to run a test for each .kat file found within the tests/reference/scripts directory.

The test first grabs the katfile, parses it and runs the model. It then grabs the reference data from the associated reference file and asserts its equality with out['test_output']. The two files must share a name, e.g., A_test.kat pairs with A_test.ref, and be in the same directory. Sub-directories are allowed. Each test is provided a unique id based on its path+filename, e.g., ref_sub1_sub2_A_test.

Adding a new reference test involves the following:

  1. create a katfile which produces a subset of the desired output using the sweep command
  2. saving the output to a .ref file of the same name (most simply using np.savetxt())
  3. placing both files in tests/reference/scripts or an appropriate sub-directory

TODOs:

  • handle exceptions; ParseException, RunException, ReadFileException(?)

Discussion:

  • grab python functions in the scripts directory which return runnable models (since not everything can be done with only katscript)
  • implement relative and absolute tolerance configurations (or define statically)

Merge request reports