Junit report displays/counts multiple tests as one test
Summary
There is some sort of parsing error where multiple test cases with similar names and/or report files that have single testsuites
tags with multiple testsuite
tags are counted as being one test. I found this under CE 11.11.8 but created an example project.
CASE 1
Resolved with !41918 (diffs)
To reproduce: Submit this xml as a junit report, and view the output in a merge request.
<?xml version="1.0" encoding="utf-8"?>
<testsuites name="unittests.dll" tests="2" failures="0" time="0.36000510000000024">
<testsuite name="unittests.dll" tests="3" skipped="0" failures="0" errors="0" time="0.36000510000000024" timestamp="2019-09-13T 23:30:12Z" hostname="executor://nunit3testexecutor/">
<testcase classname="A" name="A(.)" file="UnitTests.dll" time="1E-07" />
<testcase classname="A" name="A(!)" file="UnitTests.dll" time="1E-07" />
<testcase classname="A" name="A( )" file="UnitTests.dll" time="1E-07" />
</testsuite>
</testsuites>
Example Project
Here is a merge showing the issue.
Siphonophora/junit-test-parsing-error!1
What is the current bug behavior?
The test widget displays 'Test summary contained no changed test results out of 1 total test'
What is the expected correct behavior?
We should see 'Test summary contained no changed test results out of 3 total test'
CASE 2
Steps to reproduce
In a Typescript monorepo, I activated it in Gitlab CI:
run-tests:
stage: build
script:
- yarn ci_test
artifacts:
when: always
paths:
- packages/**/test-results.xml
reports:
junit:
- packages/**/test-results.xml
The root level yarn ci_test
iterates over the packages in the packages
folder, runs yarn test
in there and each package now contains a test-results.xml
. I get 9 files out of this, most only containing a single test doing nothing for evaluation purposes of this visualization:
Uploading artifacts...
packages/**/test-results.xml: found 9 matching files
Uploading artifacts to coordinator... ok id=509700156 responseStatus=201 Created token=<token>
I specify a top level testsuites
name reflecting the package, and testsuite
names corresponding to the test file. All this info seems to be dropped from the JUnit test reports view:
A toplevel testsuites
tag has a collection of nested testsuite
tags. Also these tags are dropped because I only see the separate testcase
occurrences in a flat list:
As mentioned earlier, if there are duplicate testcases based on name
and classname
, these are not listed.
What is the current bug behavior?
I only have a suite called run-tests
. This is the name of the stage that produced it. I am not interested in that at all!
Once I click the run-tests
suite, I also don't see my separate testsuite
collections.
What is the expected correct behavior?
Given each file has a toplevel testsuites
tag, that enumeration is what I would like to see at the top level. The toplevel set of test suites should be based on the name
tag of the toplevel testsuites
tag in each archived report file. Two files containing the same top level tag should be joined into a single top level test suite.
There are intermediate testsuite
collections, please show these too!
In there, show the separate test cases. Do not anything magic with stripping out duplicates!
CASE 3
Summary
I just found that my JUnit results are not rendered correctly. Also seen that JUnit support is currently discussed so I link this to related issues #299086 and #324620 (closed). If you consider this a duplicate, feel free to close and reference from the other issues.
Steps to reproduce
I am using Codeception in my test pipeline and it can export JUnit XML report. Here is an example test run:
Here is how gitlab shows them:
I have 4 failures and 34 errors, gitlab only shows 3 failues and 3 errors.
The problem here is that the name
property in the JUnit file is considered unique by the gitlab parser.
However in my project there are many tests with the same function name in different classes.
Also the same test is executed with different input data so multiple instance of class
-name
can exist with different value in feature
. See the following extraction from JUnit XML as an example:
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="acceptance" tests="11" assertions="154" failures="0" errors="0" time="26.059122">
<testcase file="/app/tests/acceptance/comm/NewsletterCest.php" name="testUnsubscribeNewsletter" class="tests\acceptance\comm\NewsletterCest" feature="test unsubscribe newsletter | 1" assertions="4" time="1.031930"/>
<!-- --- 8< --- -->
</testsuite>
<testsuite name="acceptancejs" tests="140" assertions="1375" failures="4" errors="34" time="3490.386799">
<testcase file="/app/tests/acceptancejs/ord/method_of_payment/BillPaymentMethodCest.php" name="testShopOrderWithPaymentMethod" class="tests\acceptancejs\ord\method_of_payment\BillPaymentMethodCest" feature="test shop order with payment method | 0,0,0,"without_popup"" assertions="17" time="26.190466"/>
<testcase file="/app/tests/acceptancejs/ord/method_of_payment/BillPaymentMethodCest.php" name="testShopOrderWithPaymentMethod" class="tests\acceptancejs\ord\method_of_payment\BillPaymentMethodCest" feature="test shop order with payment method | 0,0,0,"with_popup"" assertions="11" time="9.214829"/>
<testcase file="/app/tests/acceptancejs/ord/method_of_payment/BillPaymentMethodCest.php" name="testShopOrderWithPaymentMethod" class="tests\acceptancejs\ord\method_of_payment\BillPaymentMethodCest" feature="test shop order with payment method | 1,0,0,"with_popup"" assertions="11" time="9.220477"/>
<testcase file="/app/tests/acceptancejs/ord/method_of_payment/BillPaymentMethodCest.php" name="testShopOrderWithPaymentMethod" class="tests\acceptancejs\ord\method_of_payment\BillPaymentMethodCest" feature="test shop order with payment method | 0,1,0,"with_popup"" assertions="11" time="9.532151"/>
<testcase file="/app/tests/acceptancejs/ord/method_of_payment/BillPaymentMethodCest.php" name="testShopOrderWithPaymentMethod" class="tests\acceptancejs\ord\method_of_payment\BillPaymentMethodCest" feature="test shop order with payment method | 0,0,1,"with_popup"" assertions="4" time="28.927803">
<error type="Facebook\WebDriver\Exception\TimeoutException">BillPaymentMethodCest: Test shop order with payment method | 0,0,1,"with_popup"
Facebook\WebDriver\Exception\TimeoutException: Waited for 20 secs but text 'Vielen Dank für Ihre Bestellung.' still not found
[...]
</error>
</testcase>
<!-- --- 8< --- -->
</testsuite>
</testsuites>
Results of GitLab environment info
I am using Gitlab 13.1.4-ce.0
.