History of last 10 test failures in JUnit report
Problem(s) to solve
- Today when a test fails it is very hard to determine quickly if the test has a history of failure indicating it could be flakey or if it is the first time it has failed.
This problem was originally defined at &1875 (closed):
GitLab CI/CD shows job log information, so you can dig into why tests fail, but it requires humans reading a bunch of text. Most languages/frameworks are able to output JUnit XML (or xUnit, NUnit, PHPUnit, etc.)
The user needs a way to more efficiently digest JUnit-style XML information so that they can find out why tests fail quicker.
For Sasha and Devon
- When a test fails, I can easily see within that CI view the history of the test, so I have the context to make a judgement call on if the test is flakey and can be skipped or needs fixed.
- When a test run completes, I can easily within that CI view the historical run times of passed tests, so that I can optimize the parallelization of my CI file to run tests faster. ** *MVC version but we can extend this later to auto optimize or at least suggest *
This issue will focus on the scope of:
A historical view which showcases unit test information across successful pipelines for the default branch .
Preferably this view will be flexible and thus usable beyond just Junit. It is, however, acceptable for an MVC to start off with a single format if need be.
Permissions and Security
What does success look like, and how can we measure that?
- A user can click into a test case to see the last n runs of the test (or at least what we have artifacts for) within the CI view to see pass/fail and click for further details of that test run.
- Clicks are recorded to capture success metrics
- This new feature will be successful if we see 25% or more users who view the JUnit report page click on a test history link.