Investigate "Known Issues" feature of SQUAD (mark tests as "allowed to fail")

Issue Background

SQUAD offers the possibility to mark tests as has_known_issues. A test also features the field issues. We would like to know if this can be used to marks tests, e.g. to be able to identify them as unimportant or known to fail.

Issue description

Find out how to add a known issues and what it actually means for the SQUAD reporting. This should be documented somewhere, possibly together with a script or tool, that shows how to mark known issues.

Acceptance criteria

  • It is documented how to create a known issue
  • A small script is provided, that shows how to create a new known issue
  • The effect on the test reports of known issues has been clarified and documented
  • It is clarified if the known issues feature in SQUAD can be used to mark tests as allowed to fail

Outcome

  • Known issues can be created using the administration panel:
    • Required fields:
      • test suite and test name: <suite-name>/<test-name>, e.g. spectre-meltdown-checker-test/CVE-2018-3640)
      • One or more group/project/environment combinations: <group-name>/<project-name> - <environment-name>, e.g. linux-cip/linux-6.1.y-cip - arm_renesas_shmobile_defconfig_r8a7743-iwg20d-q7
    • The known issues tab gives an overview of tests with known issues (see e.g. here)
    • Known issues are not registered at previously reported tests. E.g. this test has known issues, but is not registered.
    • Test result reporting, after a known issue has been created, are marked as xfail. See for example this result
Edited by Sietze van Buuren