Skip to content

Pytest: add `--on-unknown-regression-files` and apply it in the CI

Arvid Jakobsson requested to merge arvid@regtest-integrity-checker into master

Context

Make the CI complain if there are unused python regression trace files.

This MR adds a new option --on-unknown-regression-files to the pytest suite, modeled directly on the tezt equivalent.

Consider its help (from poetry run pytest --help):

  --on-unknown-regression-files={warn,ignore,fail,check,delete}
                        How to handle regression test outputs that are not
                        declared by any of the collected test. MODE should be either
                        'warn', 'ignore', 'fail' or 'delete'.
                        If set to 'warn', emit a warning for unknown output files.
                        If set to 'ignore', ignore unknown output files.
                        If set to 'fail', terminate execution with exit code 1 and
                        without running any further action when unknown output files
                        are found.
                        If set to 'check', terminate execution with exit code 1 and
                        without running any further action when unknown output files
                        are found. If no unknown output files are found, run no further
                        action and terminate execution with exit code 0.
                        If set to 'delete', delete unknown output
                        files. To check which files would be deleted, run with this
                        option set to 'warn'.
                        Note that outputs that belong to uncollected tests (i.e. tests
                        not specified on the command-line) will be considered as
                        unknown. Therefore, setting any value than 'ignore' gives
                        confusing results unless a full test directories
                        (i.e. 'tests_alpha' or 'tests_XXX') are specified on the
                        command-line.

In the second commit I run it in the CI in mode check. It currently fails, waiting on !6576 (merged).

Manually testing the MR

  • Run poetry run pytest --on-unknown-regression-files check tests_python, observe that by default it will complain about unknown files.
  • Run poetry run pytest --on-unknown-regression-files warn tests_python: it will complain and then run the test suite
  • Run poetry run pytest --on-unknown-regression-files ignore tests_python: it will just run the test suite
  • Run poetry run pytest --on-unknown-regression-files fail tests_python: it will fail if there are unknown regression files. run the test suite
  • Run poetry run pytest --on-unknown-regression-files delete tests_python: will remove any trace files. you can see that it works by running with check just after.

Checklist

  • Document the interface of any function added or modified (see the coding guidelines)
  • Document any change to the user interface, including configuration parameters (see node configuration)
  • Provide automatic testing (see the testing guide).
  • For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
  • Select suitable reviewers using the Reviewers field below.
  • Select as Assignee the next person who should take action on that MR
Edited by Arvid Jakobsson

Merge request reports