Unit Test Reports: Incorrect Duration When Running Tests With Multiple Workers
Summary
In the GitLab Test Report, the total time duration is calculated by the sum of all tests and does not take into consideration the total time reported in the JUnit report.
This is problematic due to the fact that when you run tests with multiple workers, it's not simply a sum; therefore, the total time from the junit report should be taken instead.
For example, if you have two tests where each one is run by a separate worker and each worker finishes the test suite in 1s while running in parallel, that means the test suite finished in 1s, but when you sum up just the test times, it's 2 seconds, which is incorrect.
Steps to reproduce
- Create a new project
- Create a .gitlab-ci.yml file with at least one test stage using: https://docs.gitlab.com/ee/ci/quick_start/
- Under
scripts:
in the yml file, introduce a command to run the tests using multiple workers (e.g for Python by including the-n workers
flag). - Introduce test report functionality by following this guide: https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html
Example Project
- Created a folder called
sample_tests
containing 2 test files of the same tests. The tests just sleep for 10 seconds each and then pass:
import time
def test_one():
time.sleep(10)
assert True
- The .gitlab-cy.yml file for the sample_tests pipeline:
test-sample:
stage: test
script:
- pytest -n 3 sample_tests --junitxml=test-reports/sample_tests_report.xml
artifacts:
when: always
paths:
- tests/test-reports/sample_tests_report.xml
reports:
junit: tests/test-reports/sample_tests_report.xml
- Created a Merge Request and captured the conflicting test report's and the junit report's durations
What is the current bug behavior?
The duration as shown in the job logs and the generated test report does not match the duration displayed in the 'Tests' tab of the corresponding job. The latter is displayed as the sum of each individual test's duration.
What is the expected correct behavior?
The duration displayed in the pipeline's 'Test' tab must match the actual duration taken by the tests to run, as is displayed in the test report xml file.
<?xml version="1.0" encoding="utf-8"?>
<testsuites>
<testsuite name="pytest" errors="0" failures="0" skipped="0" tests="2" time="11.893" timestamp="2024-01-16T10:47:39.571510" hostname="25e3eb1dfddc">
<testcase classname="sample_tests.test_one" name="test_a" time="10.088" />
<testcase classname="sample_tests.test_two" name="test_b" time="10.095" />
</testsuite>
</testsuites>