Leftover file after test / considerations on files and plot testing
The new 'repeat' BZ plot feature is very nice. However the test_bz.py
test has a couple of issues:
- Currently it leaves a png file inside the source tree. It should instead use the
testdir
fixture from conftest. Doing so makes it run under/tmp/pytest...
, so we don't clutter the source tree or another directory the user cares about. - It should use the
datadir
fixture to access static (reference) files. If individual tests use__file__
to build paths, it creates difficult-to-see dependencies and it gradually becomes difficult to restructure the test suite.
There's also the question of how we test plots in general. The matplotlib image comparison function is quite interesting, but I am not sure about the success criteria for the test; for example whether the reference file needs to be manually updated if the plot's rotation or size changes. Our plotting settings are often very "fickle" in the sense that lots of stuff goes on the plot and we might change spacings etc., which would be a considerable burden if we have many tests based on image comparisons.
The test doesn't do anything with the return value of compare_images()
, nor do I think the function itself will raise an error. Thus, the test will pass even if the rendered image changes.
I unfortunately cannot think of a good way to test actual output images. The only way I know is: 1) refactor the code, splitting processing away from plotting, and unittest the processing, and 2) call the plotting code without asserting anything, so we at least see if it crashes.