Introduce a new failed-test-issues to report test failures in notes
What does this MR do and why?
This introduces a new dedicated script to report test failures in canonical test issues, similarly to how we report flakiness and slowness in canonical test issues.
This is very similar to how we report test flakiness and slowness, with the exception that instead of using a single note for all reports, we create a (collapsible) discussion and then one note per failure type. Then for each failure type, we report the failures in the relevant note, so that we have the number of reports per failure type.
The failure is also added to all related test issues (and the issues are marked as related), so that it'll ease converge towards a single canonical issue per test in the future.
Note: I didn't touch the relate-failure-issue script so that E2E reporting aren't modified (I don't want to change the expectation from SETs on triage. We can discuss later if they want to switch to this new centralized "test healthiness" reporting.
How to set up and validate locally
I made a few tries locally to see if this behaves as expected.
Examples of a nice collapsible discussion with one note per failure type, and list of reports for each failure type:
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Closes #54 (closed)