Tests must copy all files which they will modify
Table of contents
Current implementation
Several unit- and functional tests have session files and alike in the source tree. These tests inevitably modify files associated with these sessions (mostly XSystem config storage and alike) by merely executing the test.
Other tests do copy their files, but only do so once per CMake invocation rather than per test run.
Some tests do copy, but try to do so to a hardcoded path in the build tree.
Risks
Polluting the source tree - e.g. not even copying the files out of the source tree - is bad for obvious reasons. This has repeatedly resulted in Merge Requests where binary files have been comited by accident which were both test in- and output.
Not copying files per test run is equally bad, as it means that tests are not cleanly reproducible. Running the test twice in the same build tree gives different results.
Copying to hardcoded files is reproducible for single threaded runs, but running tests in parallel (e.g. sensible for the CI to cut down testing times by a huge factor!) still doesn't work.
Expected improvement
-
Definition where temporary files should exists -
Helper for this temp handling exists -
Tests which potentially write to files, allocate a temporary folder within the test executable. -
All files which are even potentially modified are copied into this folder. -
The path of the temporary folder is assigned by the OS or derived from a process specific attribute such as the process id.