What tests are problematic
Problem Statement
We think that customers have problems identifying tests that are "problematic". These are tests that may fail intermittently (are flaky), are slow either when passing or failing, are lumped together inefficiently , etc. This leads to long running pipelines and slow feedback cycles for developers and inefficient use of runners which can be a budget concern for VPs / Directors.
Reach
We think this impacts a large amount of our customers, around 50%, so score this a 6.0.
Impact
We think we can create a medium impact for both personas mentioned above so score this a 1.0.
Confidence
We have high confidence this is a problem but low confidence we know which part of the problem to tackle first so score this with medium confidence, 80%.
Effort
We think this would take 1 milestone (or month) for a PM, Designer, FE and BE to build an MVC that shows improvement. We score this as (1 month x 4 people) -> 4.0
Overall score - 1.2
( Large Percentage Reach (6.0) x Medium Impact (1.0) x Medium Confidence (80%) ) / Effort (4.0) = 1.2 Rice Score
Status
- We fielded a survey in early 2021 to better understand why users were not uploading test data that would drive these features.
- The majority (58%) of users simply did not know about the test report feature which would drive any flaky test identification.
- James is conducting follow-up interviews with some survey takers in May/June to discuss what problems they solve by looking at historical test data to validate next direction of this area.
- In 13.9 the MVC was added to the Test Report
- In 13.7 the MVC of test history in MRs was introduced