Test Summary on MR only presents one test failure even if multiple are present.
Summary
When a testcase produces multiple failures, these are reported with the same test case name, but different file and id locations, together with a different failure reason. In the GitLab UI these only show up as a single failure. Essentially the name of the testcase is used as a discriminator and all other testcase messages with the same name are hidden.
Maybe it was the intent to make the single visible testcase failure into an expandable list of the failures associated with that testcase, but this is not done at present.
Steps to reproduce
Return a JUnit XML file that has testcases containing the same name. Only one will turn up in the GitLab UI. For example:
<?xml version='1.0' encoding='UTF-8'?>
<testsuites errors="0" failures="2" skipped="0" tests="2">
<testsuite errors="0" failures="2" hostname="me.gerph.org" id="1" name="xml-lint" skipped="0" tests="2" timestamp="2019-01-18T08:07:00">
<properties>
<property name="CI_BRANCH" value="" />
<property name="CI_BRANCH_VERSION" value="0.0.master.0" />
<property name="CI_PROJECT_NAME" value="riscos-prm" />
<property name="CI_PROJECT_VERSION" value="0.0..1" />
<property name="CI_SHA" value="b9cf36187b24d0ddc9114820c1766a488c3ef6cd" />
<property name="CI_SHORT_BRANCH" value="" />
<property name="Linter" value="xmllint" />
<property name="SOURCE_DATE_EPOCH" value="1547798639" />
<property name="xmllint version" value="20901" />
</properties>
<testcase classname="lint" file="src/style/input.xml" line="15" name="Lint src/style/input.xml" time="0.259">
<failure message="Element section content does not follow the DTD, expecting (p | subsection | subsubsection | category | swi-definition | service-definition | upcall-definition | message-definition | command-definition | sysvar-definition | error-definition | entry-definition | vector-definition | extended-example | import)*, got (p fixme )" type="validity error"></section>
^
</failure>
</testcase>
<testcase classname="lint" file="src/style/input.xml" line="21" name="Lint src/style/input.xml" time="0.039">
<failure message="Element image does not carry attribute width" type="validity error"><image type="png" src="colourdbox.png" />
^
</failure>
</testcase>
</testsuite></testsuites>
Example Project
gerph/unhelpful-junitxml-handling!2
- shows the failure.
What is the current bug behavior?
GitLab UI shows: example found 1 failed test result out of 1 total test
What is the expected correct behavior?
GitLab UI shows: example found 2 failed tests result out of 2 total tests
(or possibly 2 failures in 1 test
, if the testcases are intended to be merged by name, but the failures reported separately)
Output of checks
This bug happens on GitLab.com
Possible fixes
-
https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/reports/test_case.rb#L23
- I think this is where the classname and the name of the test become the keys for the result. It should probably include the file and line. Although even that might be wrong if there are multiple failures on a given line, which implies that the check should also include the detail of the failure message maybe - although it has the further problem that if the file and line were used as discriminators, moving code would then show up as a fix for one error and another one raised. This implies to me that trying to do this clever comparator stuff is actually not practical in the real world.