Introduce weekly flaky test report
Purpose
This is to capture proposed item in EP team meeting to introduce weekly flaky test report similar to the weekly triage report or the bug report that's used by development teams.
The purpose of this report is to allow better surfacing of newly identified flaky tests in an overview format, and make sure every flaky test is captured in an issue. The report would group all the flaky test issues that came up recently and use that report to communicate with the Engineering Department and encourage actions.
Proposals
Option 1: Modify EP team's existing weekly triage report (for example this) to include a section for flaky tests. These would mean issues labeled with flaky-test::xxx
Option 2: Create a policy for generating a separate report exclusively for flaky tests. As a next step we could even organize these tests into group sections and tag the corresponding team EM.
Option 3: Use the existing flaky test report in Sisense https://app.periscopedata.com/app/gitlab/888968/EP---Flaky-tests
Update after getting feedback from the discussions below: Since each group already receives summary of flaky tests assigned to their group in each week's triage report (for example this), what's more appropriate to address here are 2 things
- Create flaky test issues automatically based on flaky test report
- Map rspec tests to groups
- When finished with these we will achieve the goal of summarizing flaky tests for each group. And the EP team can just include the flaky test count per group in the team's weekly triage report (example: this week's https://gitlab.com/gitlab-org/quality/triage-reports/-/issues/9859)