Unit Test Report failures should cause the job/pipeline to fail

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

The Unit Test Report documentation says:

It is very common that a CI/CD pipeline contains a test job that verifies your code. If the tests fail, the pipeline fails and users get notified.

(emphasis mine)

But that's not what actually happens in GitLab CI — failures in the Unit Test Report don't seem to affect the success or failure of a job/pipeline. The documentation seems inconsistent with the actual behavior.

Steps to reproduce

1. Create .gitlab-ci.yml:

   test-behat:
       script: echo running behat
       artifacts:
           reports:
               junit: default.xml

   and default.xml:

   <?xml version="1.0" encoding="UTF-8"?>
   <testsuites name="default">
     <testsuite name="Tika" tests="1" skipped="0" failures="1" errors="0" time="1">
       <testcase name="Tika can extract text from an image via OCR" classname="Tika" status="failed" time="1" file="features/Tika.   feature">
         <failure message="Then I should see &quot;HTTP/1.1 200 OK&quot;: Expected output to match &quot;HTTP/1.1 200 OK&quot;, but    it didn't.  Actual output: &quot;HTTP/1.1 422 Unprocessable Entity&#13;&#10;Date: Thu, 31 Oct 2024 13:37:15 GMT&#13;&#10;   Content-Length: 0&#13;&#10;&#13;&#10;&quot; (Exception)"></failure>
       </testcase>
     </testsuite>
   </testsuites>

2. Commit and push changes to a GitLab project
3. The pipeline runs, and the job and pipeline unexpectedly succeed

Example Project

https://gitlab.com/smokris/test-ci-junit-fail

What is the current bug behavior?

Despite the Unit Test Report indicating a failed test — <testsuite failures="1"> and <testcase status="failed">and` — the job and pipeline succeed.

What is the expected correct behavior?

As mentioned in the documentation: "If the tests fail, the pipeline fails"

Relevant logs and/or screenshots

https://gitlab.com/smokris/test-ci-junit-fail/-/jobs/8244366038:

Output of checks

This bug happens on GitLab.com