History of last 10 test failures in JUnit report
Problem(s) to solve
- Today when a test fails it is very hard to determine quickly if the test has a history of failure indicating it could be flakey or if it is the first time it has failed.
This problem was originally defined at &1875 (closed):
GitLab CI/CD shows job log information, so you can dig into why tests fail, but it requires humans reading a bunch of text. Most languages/frameworks are able to output JUnit XML (or xUnit, NUnit, PHPUnit, etc.)
The user needs a way to more efficiently digest JUnit-style XML information so that they can find out why tests fail quicker.
Intended users
Further details
Use Cases/outcomes
For Sasha and Devon
- When a test fails, I can easily see within that CI view the history of the test, so I have the context to make a judgement call on if the test is flakey and can be skipped or needs fixed.
- When a test run completes, I can easily within that CI view the historical run times of passed tests, so that I can optimize the parallelization of my CI file to run tests faster. ** *MVC version but we can extend this later to auto optimize or at least suggest *
Proposal
This issue will focus on the scope of:
A historical view which showcases unit test information across successful pipelines for the default branch .
Preferably this view will be flexible and thus usable beyond just Junit. It is, however, acceptable for an MVC to start off with a single format if need be.
Permissions and Security
Documentation
Testing
What does success look like, and how can we measure that?
Acceptance Criteria
- A user can click into a test case to see the last n runs of the test (or at least what we have artifacts for) within the CI view to see pass/fail and click for further details of that test run.
- Clicks are recorded to capture success metrics
Success Metrics
- This new feature will be successful if we see 25% or more users who view the JUnit report page click on a test history link.
Links / references
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.