Skip to content

TOP-847: Use Allure for test reporting

Sander Ploegsma requested to merge top-847-allure into main

This updates our project to generate a test report using Allure Report. Main features of Allure are:

  • The ability to add metadata to tests, suites, etc
  • The ability to annotate methods as test steps, so that they become browsable as a tree in the report
  • The ability to attach files (images, CSV, videos, PDFs and more) as test evidence to individual test results
  • The ability to record test environment information
  • The ability to record test executor information

Along with some other features that make the results much more easy to browse. Here's an example report generated from our CI/CD pipeline: https://ska-telescope.gitlab.io/-/ska-low-csp-test/-/jobs/6240936474/artifacts/build/reports/allure/index.html

This features:

  • Allure metadata on all existing tests.
  • The SPEAD metadata attached as a CSV file on the LOW-ITF integration test.
  • Some example environment properties containing the subsystem software versions. There are probably more properties to add, but these should serve as a good example of how to set them.
  • Executor information pulled from GitLab CI/CD variables

Some things to note:

  • I grouped the integration and verification tests into a single CI/CD job called tests:regression, since splitting it up into multiple jobs isn't really needed for now.
  • I updated the CI/CD job to properly handle test failures: tests classified as error (=failure in the setup or teardown) are not permitted, while tests classified as failure (=assertion error in the test method itself) will cause the pipeline to warn but not fail. The test report is generated in all cases.
  • I moved the unit test files next to their respective modules in src. It's a bit unconventional but it works, and gives a clear separation of concerns.
  • I moved the obsState assertions out of the SubarrayUnderTest class, mainly because they were giving me headaches when the state changes were lagging a bit. IMO this is fine, since the tests that are interested in the underlying state changes can just assert on them themselves.
  • To prevent a lot of duplication between the @allure decorators and the docstrings I removed all docstrings from the test files, and added pylint:disable rules for them. I wish there was a way to automatically disable these rules for all files in the tests/ directory but I couldn't find one.
Edited by Sander Ploegsma

Merge request reports