port rtest to pytest
The Graphviz test suite is spread amongst the subdirectories ci/, tests/ and rtest/. Tests are currently orchestrated via Pytest. Within this menagerie, rtest/rtest.py is a former Python (or even Perl?) script that has been dragged into the Pytest world against its will. As a result it is not particularly Pythonic and its failures are reported quite cryptically:
Test foo:bar : == Failed == baz.png
"OK, cool, but what were the differences between the actual output and the reference output?"
My (partially uninformed) summary of what is happening here is that rtest.py was once expected to be run locally. Hence why it outputs minimal information and instead dumps an informative diff as an HTML page for you to load. Of course this HTML page is inaccessible when it is generated in the CI environment. The CI environment is now the main way Graphviz developers execute the test suite. That is, the ground has shifted underneath rtest.py making its assumptions/expectations no longer accurate.
This issue is a proposal to port/rewrite rtest.py (or more accurately, rtest/tests.txt) into Pytest. This should allow (1) more detailed failure feedback, (2) more fine-grained test case failures, and (3) the ability to run rtest.py test cases in parallel. Note that this will require undoing some assumptions, e.g. that os.getpid()
is a valid way to construct a temporary filename.
Related to #195 (closed).