Skip to content

Support relating test failures from JUnit Reports generate by gotestsum

John McDonnell requested to merge jmd/support-gotestsum-formatted-junit-reports into main

What does this MR do and why?

It may be useful if the gem could be used outside of the context of a report generated by RspecJunitFormatter.
A first specific use case for this is for use with gitaly which makes use of a library gotestsum to generate a JUnit style report.

Various tooling may produce more/less fields and data in the report as the case may be. Therefore a large part of supporting this will be to handle cases where a a report may or may not have a specific field. Note for example here how the file is not present in the Golang style report, which causes a series of failures if we attempt to process such a report today.

This MR adds several layers of hardening to the tooling to avoid raising exceptions when processing these Junit reports do not include fields that we may have previously assumed would be present in the JSON formatted reports.

Sample output of reports may look like the following:

Golang Style Junit Report 
<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="148" failures="7" errors="0" time="27.178139">
    <testsuite tests="148" failures="7" time="23.228000" name="gitlab.com/gitlab-org/gitaly/v16/internal/backup" timestamp="2023-10-10T10:20:47+01:00">
        <properties>
            <property name="go.version" value="go1.21.1 darwin/arm64"></property>
        </properties>
        <testcase
                classname="gitlab.com/gitlab-org/gitaly/v16/internal/backup"
                name="TestManager_Create/hooks"
                time="0.660000">
            <failure message="Failed" type="">=
                == RUN   TestManager_Create/hooks&#xA;    backup_test.go:137: &#xA;        &#x9;Error Trace:&#x9;/Users/john/dev/gdk/gitaly/internal/backup/backup_test.go:137&#xA;        &#x9;            &#x9;&#x9;&#x9;&#x9;/Users/john/dev/gdk/gitaly/internal/backup/backup_test.go:169&#xA;        &#x9;Error:      &#x9;Received unexpected error:&#xA;        &#x9;            &#x9;open /private/tmp/gitaly-2357346906/794094028/storages.d/default/@hashed/c4/7a/c47adcd3b006e1847beebb637395dff170ecaf846f3cc0de8ffeb0d22ac63aa9.git/custom_hooks/pre-commit.sample: no such file or directory&#xA;        &#x9;Test:       &#x9;TestManager_Create/hooks&#xA;--- FAIL: TestManager_Create/hooks (0.66s)&#xA;
            </failure>
        </testcase>
    </testsuite>
</testsuites>
Ruby Style Junit Report 
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="rspec" tests="1" skipped="0" failures="1" errors="0" time="44.609179" timestamp="2023-10-10T10:53:02+01:00" hostname="John-Macbook-M1.local">
    <properties>
        <property name="seed" value="61264"/>
    </properties>
    <testcase
            classname="qa.specs.features.browser_ui.3_create.snippet.clone_push_pull_project_snippet_spec"
            name="Create Version control for project snippets clones, pushes, and pulls a project snippet over HTTP, edits via UI"
            file="./qa/specs/features/browser_ui/3_create/snippet/clone_push_pull_project_snippet_spec.rb"
            time="44.600484">
        <failure
                message="expected &quot;8ad7569 Update snippet&quot; to include &quot;Update snippetsss&quot;"
                type="RSpec::Expectations::ExpectationNotMetError">
            Failure/Error: expect(repository.commits.first).to include &apos;Update snippetsss&apos;
            expected &quot;8ad7569 Update snippet&quot; to include &quot;Update snippetsss&quot;
            ./qa/specs/features/browser_ui/3_create/snippet/clone_push_pull_project_snippet_spec.rb:67:in `block (4 levels) in &lt;module:QA&gt;&apos;
            ./qa/scenario/actable.rb:16:in `perform&apos;
            ./qa/git/repository.rb:32:in `block (2 levels) in perform&apos;
            ./qa/git/repository.rb:32:in `chdir&apos;
            ./qa/git/repository.rb:32:in `block in perform&apos;
            ./qa/git/repository.rb:31:in `perform&apos;
            ./qa/specs/features/browser_ui/3_create/snippet/clone_push_pull_project_snippet_spec.rb:62:in `block (3 levels) in &lt;module:QA&gt;&apos;
        </failure>
        <system-out>
            [[ATTACHMENT|/Users/john/dev/gdk/gitlab/qa/tmp/qa-test-2023-10-10-10-53-08-bd1c262e44fe6ac9/create_version_control_for_project_snippets_clones_pushes_and_pulls_a_project_sn_2023-10-10-10-53-46.995.png]]
        </system-out>
    </testcase>
</testsuite>

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

  • From gdk/gitaly modify some code/tests to induce a failure and run make test-go
  • Assuming gdk is running on gdk.test:3000
  • Create a test project gitlab-org/gitaly on gdk.
  • CI_PROJECT_NAME=gitaly-gdk CI_PIPELINE_URL=http://gdk.test:3000/gitlab-org/gitaly/-/pipelines/607 CI_JOB_URL=http://gdk.test:3000/gitlab-org/gitaly/-/jobs/464 CI_API_V4_URL=http://gdk.test:3000/api/v4 bundle exec relate-failure-issue --project gitlab-org/gitaly --input-files test-report-go.xml --token '<ACCESS_TOKEN>'

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by John McDonnell

Merge request reports