Do not aggregate TestReport when using matrix job
requested to merge morefice/test-reports-screen-should-include-matrix-id-as-part-of-job-name-to-differentiate-matrix into master
Ref: #296814 (closed)
What does this MR do and why?
This MR fixes a typebug where all parallel matrix jobs are aggregated in 1 job.
Why are we doing this?
When using parallel jobs we aggregate all specs using build#group_name when building our TestReport.
This makes sense for gitlab.com
where all jobs running are parallelized to speed things up, as you can see on this page.
However the team decided to not aggregate multiple TestSuite
results when users are using a matrix
job so users can generate test reports
correctly using matrix jobs.
Screenshots or screen recordings
Ressource | Before | After |
---|---|---|
Pipeline |
||
Merge Request |
Testing locally
- Create a job using the
gitlab-ci.yml
provided below - Verify test report result is not being aggregated on
http://localhost:3000/root/<PROJECT_ID>/-/pipelines/<PIPELINE_ID>/test_report
- Verify test report result is not being aggregated on the MR widget
gitlab-ci.yml
matrix:
parallel:
matrix:
- ID: ["1", "2"]
script:
- ls
artifacts:
when: always
paths:
- report-${ID}.xml
reports:
junit:
- report-${ID}.xml
report-1.xml
<?xml version="1.0"?>
<testsuite errors="0" failures="1" tests="2" name="from cppunit">
<testcase classname="EventConstsTest" name="TestRecFitQuality">
<failure message="assertion failed" type="Assertion">assertion failed
One type of problem
Line: test.cc:23</failure>
</testcase>
<testcase classname="EventConstsTest" name="TestRecTrackConst"/>
</testsuite>
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Max Orefice