Skip to content

Follow-up: improve test results handling for NUNIT type

The following discussion from !95 (merged) should be addressed:

  • @coderabbitai started a discussion: (+2 comments)

    💡 Codebase verification

    Test results handling needs to be updated for NUNIT type

    The test script currently only handles result transformation for 'JUNIT' type but doesn't have specific handling for 'NUNIT' type. This needs to be addressed since:

    • The script checks for TESTING_TYPE == 'JUNIT' in multiple places
    • NUNIT results are left in their native format without transformation
    • The new configuration explicitly sets TESTING_TYPE: NUNIT

    Required changes:

    • Remove JUNIT-specific conditionals or update them to handle NUNIT type
    • Ensure GitLab can parse native NUNIT results, or add NUNIT result transformation
    • Update result parsing in the final grep test-run command to match NUNIT format
    🔗 Analysis chain

    Verify test results parsing with NUNIT

    The addition of TESTING_TYPE: NUNIT explicitly defines the testing framework. Ensure that the test.sh script correctly handles NUNIT test results.

    Let me verify the test result handling further by checking for any NUNIT-specific transformations or configurations.

    Let me check the actual test result handling in the test script to see how NUNIT results are processed.

    🏁 Scripts executed

    The following scripts were executed for the analysis:

    Script:

    #!/bin/bash
# Description: Check if test.sh handles NUNIT test type

# Check test.sh for NUNIT handling
if [ -f "./ci/test.sh" ]; then
  echo "Checking test.sh for NUNIT handling..."
  grep -A 5 "TESTING_TYPE" "./ci/test.sh"
fi

    Length of output: 1048

    Script:

    #!/bin/bash
# Check for NUNIT test result handling configurations
echo "Checking for NUNIT configurations..."
rg -l "NUNIT|nunit" -g "!*.xml"

# Check contents of CI scripts directory
echo -e "\nListing CI scripts directory:"
ls -la ci/

# Check for test result transformation files
echo -e "\nChecking transformation files:"
ls -la ci/nunit-transforms/ 2>/dev/null || echo "No nunit-transforms directory found"

    Length of output: 1152

    Script:

    #!/bin/bash
# Check test.sh content for test result handling
echo "Checking test result handling in test.sh:"
cat ci/test.sh

    Length of output: 2238

I'm not sure about this one. Feel free to provide more details and suggest a code change to properly achieve this.

@gableroux, certainly! I'd be happy to provide more details and suggest code changes to properly handle NUNIT test results in your pipeline.

Issue Explanation:

In your .gitlab-ci.yml, you've set TESTING_TYPE: NUNIT for your test jobs:

test-playmode:
  <<: *test
  variables:
    TEST_PLATFORM: playmode
    TESTING_TYPE: NUNIT

test-editmode:
  <<: *test
  variables:
    TEST_PLATFORM: editmode
    TESTING_TYPE: NUNIT

However, in your ci/test.sh script, the handling for test results and artifacts is primarily designed for when TESTING_TYPE is set to 'JUNIT'. This means that when TESTING_TYPE is 'NUNIT', the script doesn't process the test results in a way that GitLab CI can interpret and display in the pipeline's test reports.

Suggested Code Changes:

  1. Modify ci/test.sh to Handle NUNIT Results:

Update your ci/test.sh script to process NUNIT test results and convert them to JUnit format if necessary. Here's how you can modify the script:

#!/bin/bash
set -euo pipefail

echo "Running Unity tests for $TEST_PLATFORM with $TESTING_TYPE testing type"
echo "Using Unity version $UNITY_VERSION"

# Run Unity tests
unity-editor \
  -batchmode \
  -projectPath "$UNITY_DIR" \
  -runTests \
  -testPlatform "$TEST_PLATFORM" \
  -testResults "$UNITY_DIR/$TEST_PLATFORM-results.xml" \
  -logFile /dev/stdout

UNITY_EXIT_CODE=$?

# Check Unity exit code
if [ $UNITY_EXIT_CODE -eq 0 ]; then
  echo "Run succeeded, no failures occurred"
elif [ $UNITY_EXIT_CODE -eq 2 ]; then
  echo "Run succeeded, some tests failed"
else
  echo "Run failed with exit code $UNITY_EXIT_CODE"
fi

# Process test results for NUNIT
if [ "$TESTING_TYPE" == "NUNIT" ]; then
  echo "Processing NUNIT test results"

  # If GitLab requires JUnit format, convert the NUNIT results
  # You can use an XSLT transformation or a tool like NUnit2Report
  # Here's an example using xsltproc and an XSLT stylesheet

  # Ensure xsltproc is installed
  apt-get update && apt-get install -y xsltproc

  # Convert NUNIT results to JUnit format
  xsltproc "$CI_PROJECT_DIR/ci/nunit-transforms/nunit3-junit.xslt" \
    "$UNITY_DIR/$TEST_PLATFORM-results.xml" \
    > "$UNITY_DIR/$TEST_PLATFORM-junit-results.xml"
fi

# Exit with the Unity exit code
exit $UNITY_EXIT_CODE
  1. Adjust .gitlab-ci.yml to Collect Test Reports:

Update your .gitlab-ci.yml to specify the test reports for GitLab CI. Modify the artifacts section in your test jobs to include the JUnit test reports:

.test: &test
  stage: build_and_test
  <<: *unity_defaults
  script:
    - chmod +x ./ci/test.sh && ./ci/test.sh
  artifacts:
    when: always
    reports:
      junit: $UNITY_DIR/$TEST_PLATFORM-junit-results.xml
    expire_in: 2 weeks
  tags:
    - gitlab-org
  coverage: /<Linecoverage>(.*?)</Linecoverage>/
  1. Ensure the XSLT Stylesheet is Available:

Place the nunit3-junit.xslt stylesheet in your repository at ci/nunit-transforms/nunit3-junit.xslt. You can obtain this stylesheet from various sources or create one that suits your project's needs.

Explanation:

  • Test Script Changes: By installing xsltproc and converting the NUNIT XML results to JUnit format, GitLab CI can parse and display the test results properly in the pipeline.

  • GitLab CI Configuration: Specifying the JUnit report in the artifacts section allows GitLab to display the test results in the merge request and pipeline views.

  • Why Convert NUNIT to JUnit: GitLab CI natively supports JUnit test reports for displaying test results. Converting NUNIT results to JUnit format bridges the compatibility gap.

Alternative Approach:

If you prefer not to convert the results and if GitLab CI now supports NUNIT test reports natively (you might want to check the latest GitLab documentation for this), you can adjust the artifacts to point directly to the NUNIT results:

artifacts:
  when: always
  reports:
    junit: $UNITY_DIR/$TEST_PLATFORM-results.xml
  expire_in: 2 weeks

In this case, you might not need to perform the conversion in the test script.

Let me know if this helps or if you have any questions!

Edited by Gabriel Le Breton