Add Support for Duplicate test names & classes in JUnit report feature
Problem to solve
As a developer, when I have a JUnit report file that has duplicate test name & class combinations, I want to be able to see all of the entries on the JUnit report viewing screen, so that the test counts match up, and I can continue to use programatically generated test names and classes in my JUnit report file.
Right now we are using a combination of classname
and name
to create keys for our hash that we load the JUnit report into: https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/reports/test_case.rb#L26. Because of this, if there are multiple entries in one JUnit report file with the same name and classname, the report will count them individually, but once it is parsed and loaded into GitLab, they will be squashed into one entry. This leads to test counts that seem incorrect on the test page and prevents 1:1 mapping of test report file to GitLab UI from being 100% accurate.
Intended users
User experience goal
I want to be able to see all of the entries on the JUnit report viewing screen, so that the test counts match up.
Proposal
Append to the test name (n) for every instance of a duplicate found for display on the Test Summary and Unit Test report.
Options discussed
There are many different approaches we could take to solve this problem, the following were discussed before 3. was picked.
- A boring one might be to take the line number in the JUnit report and use that as part of the composite key for the hash.
- Display an * by the test that was duplicated with a note on the page that * indicates multiple test names were found but only one displayed.
- Append to the test name (n) for every instance of a duplicate found.
Further details
This was initially reported in another issue about jest tests counts not matching up.
Permissions and Security
Documentation
- Remove any notes about why duplicate tests are not displayed correctly from the existing docs.
Availability & Testing
What does success look like, and how can we measure that?
Acceptance Criteria
- When there are duplicate instances of a test with the same suite/test name combo in separate junit files the total count of tests should match up between screens of the JUnit report.
What is the type of buyer?
This feature most applies to the Software Developer and is an enhancement to a GitLab Core feature.
Is this a cross-stage feature?
No.
Links / references
Reproduction Steps
From #231478 (closed)
Summary
When I navigate between the list of jobs and the detail of a jobs on the Test Report I see two different counts of tests which makes me question the accuracy of any of the data.
This bug is affecting only the jest
job.
Steps to reproduce
- Navigate to a project with 10s of thousands of tests like gitlab.
- Navigate to a pipeline that ran all the tests
- Click on the test report tab
- Note the number of tests
- Click on a job
Example Project
https://gitlab.com/gitlab-org/gitlab/-/pipelines/169603204/test_report
What is the expected correct behavior?
Numbers match.
What is the current bug behavior?
The numbers are different, see the images below.
Relevant logs and/or screenshots
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)