Skip to content

Do not aggregate TestReport when using matrix job

Ref: #296814 (closed)

What does this MR do and why?

This MR fixes a typebug where all parallel matrix jobs are aggregated in 1 job.

Why are we doing this?

When using parallel jobs we aggregate all specs using build#group_name when building our TestReport.

This makes sense for gitlab.com where all jobs running are parallelized to speed things up, as you can see on this page.

However the team decided to not aggregate multiple TestSuite results when users are using a matrix job so users can generate test reports correctly using matrix jobs.

Screenshots or screen recordings

Ressource Before After
Pipeline Screenshot_2022-05-30_at_14.24.29 Screenshot_2022-05-18_at_11.56.21
Merge Request Screenshot_2022-05-30_at_15.29.39 Screenshot_2022-05-30_at_15.17.36

Testing locally

  1. Create a job using the gitlab-ci.yml provided below
  2. Verify test report result is not being aggregated on http://localhost:3000/root/<PROJECT_ID>/-/pipelines/<PIPELINE_ID>/test_report
  3. Verify test report result is not being aggregated on the MR widget

gitlab-ci.yml
matrix:
  parallel:
    matrix:
      - ID: ["1", "2"]
  script:
    - ls
  artifacts:
    when: always
    paths:
      - report-${ID}.xml
    reports:
      junit:
        - report-${ID}.xml
report-1.xml
<?xml version="1.0"?>
<testsuite errors="0" failures="1" tests="2" name="from cppunit">
  <testcase classname="EventConstsTest" name="TestRecFitQuality">
    <failure message="assertion failed" type="Assertion">assertion failed
      One type of problem
Line: test.cc:23</failure>
  </testcase>
  <testcase classname="EventConstsTest" name="TestRecTrackConst"/>
</testsuite>

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Max Orefice

Merge request reports