Pytest: add `--on-unknown-regression-files` and apply it in the CI
Context
Make the CI complain if there are unused python regression trace files.
This MR adds a new option --on-unknown-regression-files
to the pytest suite, modeled directly on the tezt equivalent.
Consider its help (from poetry run pytest --help
):
--on-unknown-regression-files={warn,ignore,fail,check,delete}
How to handle regression test outputs that are not
declared by any of the collected test. MODE should be either
'warn', 'ignore', 'fail' or 'delete'.
If set to 'warn', emit a warning for unknown output files.
If set to 'ignore', ignore unknown output files.
If set to 'fail', terminate execution with exit code 1 and
without running any further action when unknown output files
are found.
If set to 'check', terminate execution with exit code 1 and
without running any further action when unknown output files
are found. If no unknown output files are found, run no further
action and terminate execution with exit code 0.
If set to 'delete', delete unknown output
files. To check which files would be deleted, run with this
option set to 'warn'.
Note that outputs that belong to uncollected tests (i.e. tests
not specified on the command-line) will be considered as
unknown. Therefore, setting any value than 'ignore' gives
confusing results unless a full test directories
(i.e. 'tests_alpha' or 'tests_XXX') are specified on the
command-line.
In the second commit I run it in the CI in mode check
. It currently fails, waiting on !6576 (merged).
Manually testing the MR
- Run
poetry run pytest --on-unknown-regression-files check tests_python
, observe that by default it will complain about unknown files. - Run
poetry run pytest --on-unknown-regression-files warn tests_python
: it will complain and then run the test suite - Run
poetry run pytest --on-unknown-regression-files ignore tests_python
: it will just run the test suite - Run
poetry run pytest --on-unknown-regression-files fail tests_python
: it will fail if there are unknown regression files. run the test suite - Run
poetry run pytest --on-unknown-regression-files delete tests_python
: will remove any trace files. you can see that it works by running withcheck
just after.
Checklist
-
Document the interface of any function added or modified (see the coding guidelines) -
Document any change to the user interface, including configuration parameters (see node configuration) -
Provide automatic testing (see the testing guide). -
For new features and bug fixes, add an item in the appropriate changelog ( docs/protocols/alpha.rst
for the protocol and the environment,CHANGES.rst
at the root of the repository for everything else). -
Select suitable reviewers using the Reviewers
field below. -
Select as Assignee
the next person who should take action on that MR
Edited by Arvid Jakobsson