JUnit XML Test Summary In MR widget
Problem to solve
It is very common that a pipeline contains a test job that will check the code. If the tests fail, the pipeline fails and users get notified. But they still want to dig into details about the failed tests.
At the moment, the only way to get more information about failed tests is to open the job output log and look for information there. This may not be easy.
People could be also be interested to see if their code is fixing some existing test that failed on the
master branch, so we can do a comparison between the source and the target branch and report differences.
baseis the report for on the commit where the source branch has been created
headis the report for the latest commit of the source branch
Add a new MR widget panel to show test results.
This panel will have a summary, showing how many tests failed and how many were fixed.
This panel will also show a list of JUnit tests:
- [red] errors that failed on
headbut succeeded on
- [red] other errors that failed on
head(existing failures, brand new tests that failed)
- [green] tests that succeded on
headbut failed on
If no comparison can be done because data for
base are not available, the panel will just show the list of failed tests for
Each entry will show details (test name, stack trace, link to the file if available). Clicking on the test name will open a modal window with details and stack trace for the given entry.
The source of the report will be a JUnit XML artifact created by the test job and managed as defined in https://gitlab.com/gitlab-org/gitlab-ce/issues/46809.
What does success look like, and how can we measure that?
Users look at test results. We can measure how many test reports will be generated, and how many reports will be accessed.
A design artifact has been addressed in https://gitlab.com/gitlab-org/gitlab-ce/issues/46178.
Values such as "class" (both in the modal as in merge request widget summary) are not always available so they will only be shown if that information is available.
Grouping will be done on a suite name. Would be constructed from Pipeline job grouping: https://docs.gitlab.com/ee/ci/pipelines.html#grouping-similar-jobs-in-the-pipeline-graph
Test summary contained no changed test results out of X total tests
Test summary contained X failed test results out of Y total tests
Test summary contained X fixed test results out of Y total tests
Test summary contained X failed results and Y fixed test results out of Z total tests
Rspec found no changed test results out of X total tests
Rspec found X failed test results out of Y total tests
Rspec found X fixed test results out of Y total tests
Rspec found X failed and Y fixed test results out of Z total tests
Extra loading state for when things are being parsed (let's overcommunicate rather than leave confusion)
Test summary results are being parsed