TOP-847: Use Allure for test reporting
This updates our project to generate a test report using Allure Report. Main features of Allure are:
- The ability to add metadata to tests, suites, etc
- The ability to annotate methods as test steps, so that they become browsable as a tree in the report
- The ability to attach files (images, CSV, videos, PDFs and more) as test evidence to individual test results
- The ability to record test environment information
- The ability to record test executor information
Along with some other features that make the results much more easy to browse. Here's an example report generated from our CI/CD pipeline: https://ska-telescope.gitlab.io/-/ska-low-csp-test/-/jobs/6240936474/artifacts/build/reports/allure/index.html
This features:
- Allure metadata on all existing tests.
- The SPEAD metadata attached as a CSV file on the LOW-ITF integration test.
- Some example environment properties containing the subsystem software versions. There are probably more properties to add, but these should serve as a good example of how to set them.
- Executor information pulled from GitLab CI/CD variables
Some things to note:
- I grouped the integration and verification tests into a single CI/CD job called
tests:regression
, since splitting it up into multiple jobs isn't really needed for now. - I updated the CI/CD job to properly handle test failures: tests classified as error (=failure in the setup or teardown) are not permitted, while tests classified as failure (=assertion error in the test method itself) will cause the pipeline to warn but not fail. The test report is generated in all cases.
- I moved the unit test files next to their respective modules in
src
. It's a bit unconventional but it works, and gives a clear separation of concerns. - I moved the
obsState
assertions out of theSubarrayUnderTest
class, mainly because they were giving me headaches when the state changes were lagging a bit. IMO this is fine, since the tests that are interested in the underlying state changes can just assert on them themselves. - To prevent a lot of duplication between the
@allure
decorators and the docstrings I removed all docstrings from the test files, and addedpylint:disable
rules for them. I wish there was a way to automatically disable these rules for all files in thetests/
directory but I couldn't find one.
Edited by Sander Ploegsma