Create a process to verify Predictive Test Selection accuracy
Context
I think the fastest way to know whether our Predictive Test Selection solution is working well is two-fold:
- Look at the overall numbers (that include false positives like flaky tests/infra failures on the full pipelines)
- Look at the specific transitions regularly to find real test selection gaps in MRs. I think this will give us insights and validate the overall numbers.
- If we have a lot of false positives (i.e. a lot of flakiness/infra failures on full backend pipelines), we'll know the overall numbers we'll see are a very pessimistic number, and that the real accuracy would be higher.
- If we have a lot of true positives, we'll have test-selection-gaprspec to fix!
Goals
- Make a process (manual at first) to verify how accurate our predictive test accuracy solution really is.
- Practice that process, and use this knowledge to automate the process
Edited by David Dieulivol