JUnit Parser ignore errors in tests
Problem to solve
Gitlab does not show errors in tests
Further details
testsuite example
<?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-failsafe-plugin/xsd/failsafe-test-report.xsd" name="personnel.ForecastListIT" time="777.326" tests="3" errors="1" skipped="0" failures="1">
<properties>
<!-- huge prop list -->
</properties>
<testcase name="shouldImportLoad" classname="personnel.ForecastListIT" time="98.306">
<error message="error message" type="org.openqa.selenium.WebDriverException">some more info
</error>
<system-out><![CDATA[some logs]]></system-out>
<system-err><![CDATA[some err]]></system-err>
</testcase>
<testcase name="shouldSetMiniumOfOperators" classname="personnel.ForecastListIT" time="280.854">
<failure message="expected:<16> but was:<3>" type="java.lang.AssertionError"><![CDATA[java.lang.AssertionError: expected:<16> but was:<3>
at personnel.ForecastListIT.shouldSetMiniumOfOperators(ForecastListIT.java:313)
]]></failure>
<system-out><![CDATA[some logs]]></system-out>
<system-err><![CDATA[some err]]></system-err>
</testcase>
<testcase name="shouldDisplayLoad" classname="personnel.ForecastListIT" time="40.013"/>
</testsuite>
in MR Gitlab show 3 tests 1 failure
but actually there are only 1 successfull test
Proposal
show not only failures, but errors too
What does success look like, and how can we measure that?
i suggest 2 possible looks
Clear: Show 1 failure, 1 error of 3 tests
Simple: Show 2 failure of 3 tests
Simple solution like this :) /gitlab/ci/parsers/junit.rb
def create_test_case(data)
if data['failure']
status = ::Gitlab::Ci::Reports::TestCase::STATUS_FAILED
system_output = data['failure']
elsif data['error']
status = ::Gitlab::Ci::Reports::TestCase::STATUS_FAILED
system_output = data['error']
else
status = ::Gitlab::Ci::Reports::TestCase::STATUS_SUCCESS
system_output = nil
end