JUnit XML Test Summary In MR widget
Problem to solve
It is very common that a pipeline contains a test job that will check the code. If the tests fail, the pipeline fails and users get notified. But they still want to dig into details about the failed tests.
At the moment, the only way to get more information about failed tests is to open the job output log and look for information there. This may not be easy.
Further details
People could be also be interested to see if their code is fixing some existing test that failed on the master
branch, so we can do a comparison between the source and the target branch and report differences.
Proposal
-
base
is the report for on the commit where the source branch has been created -
head
is the report for the latest commit of the source branch
Add a new MR widget panel to show test results.
This panel will have a summary, showing how many tests failed and how many were fixed.
This panel will also show a list of JUnit tests:
- [red] errors that failed on
head
but succeeded onbase
(new failures) - [red] other errors that failed on
head
(existing failures, brand new tests that failed) - [green] tests that succeded on
head
but failed onbase
(fixed tests)
If no comparison can be done because data for base
are not available, the panel will just show the list of failed tests for head
.
Each entry will show details (test name, stack trace, link to the file if available). Clicking on the test name will open a modal window with details and stack trace for the given entry.
The source of the report will be a JUnit XML artifact created by the test job and managed as defined in https://gitlab.com/gitlab-org/gitlab-ce/issues/46809.
What does success look like, and how can we measure that?
Users look at test results. We can measure how many test reports will be generated, and how many reports will be accessed.
Design artifact
A design artifact has been addressed in https://gitlab.com/gitlab-org/gitlab-ce/issues/46178.
Design
-
Values such as "class" (both in the modal as in merge request widget summary) are not always available so they will only be shown if that information is available.
-
Grouping will be done on a suite name. Would be constructed from Pipeline job grouping: https://docs.gitlab.com/ee/ci/pipelines.html#grouping-similar-jobs-in-the-pipeline-graph
-
Test summary contained no changed test results out of X total tests
-
Test summary contained X failed test results out of Y total tests
-
Test summary contained X fixed test results out of Y total tests
-
Test summary contained X failed results and Y fixed test results out of Z total tests
-
Rspec found no changed test results out of X total tests
-
Rspec found X failed test results out of Y total tests
-
Rspec found X fixed test results out of Y total tests
-
Rspec found X failed and Y fixed test results out of Z total tests
-
Extra loading state for when things are being parsed (let's overcommunicate rather than leave confusion)
Test summary results are being parsed