Support skipped tests in JUnit XML Report in JUnit test reports
Problem to solve
Sasha is writing unit tests along with code with results in a format GitLab can consume and report on about what passed and failed. This is great if all tests are run but Sasha is starting to write more complex tests that are sometimes skipped based on previous test results or other external factors to the test suite. Today those skipped tests are showing up as successful test executions in the GitLab reports, an incorrect signal to Sasha and the team about the reliability of their code.
Some testing frameworks allow for tests to be skipped at runtime. This is convenient for various reasons, such as if resources cannot be properly allocated or if there is a filesystem issue. Currently, if a test is skipped, the CI pipeline passes with no indication of skipped tests. While a skipped test shouldn't necessarily be a total failure, it also shouldn't be a success. It would be nice for a warning to be displayed if one or more tests are skipped at runtime. If a test case result is "skipped", the CI pipeline should pass with warnings.The original writeup for this feature was from @sethraymond, thanks!
Description
Proposal
Links to related issues and merge requests / references
Intended users
- Sasha (Software Developer) - who is seeing test results in pipeline results and merge requests.
- Simone (Software Engineer in Test) - Who is validating test coverage and effectiveness of those tests.
Further details
This enhances the JUnit reports which already has a column for skipped but is not being populated today.
A future enhancement could be to update the Test Summary Widget to include additional information about the tests skipped on the head branch compared to results on the base branch.
Proposal
- Update parsing of JUnit results to capture skipped tests as their own counted item.
- The docs indicate cases will have a entry and the test suite will have a skipped= count.
This is mentioned in the related issue.
This seems to be a spec/test for this feature, it does not seem to inlcude a test for skipped tests: https://gitlab.com/gitlab-org/gitlab/-/blob/acc05479b1723f8804579548d8d3bde2478c18fd/spec/lib/gitlab/ci/parsers/test/junit_spec.rb
This code seems to be where it creates the report - it seems to only handle pass, fail and error, not skipped. https://gitlab.com/gitlab-org/gitlab/-/blob/acc05479b1723f8804579548d8d3bde2478c18fd/lib/gitlab/ci/parsers/test/junit.rb#L48
Permissions and Security
Documentation
Availability & Testing
What does success look like, and how can we measure that?
Acceptance Criteria
- A sample project that outputs skipped tests counts those correctly on the JUnit report
- The MR widget still functions as expected before the change
What is the type of buyer?
- This is GitLab Core functionality.
Is this a cross-stage feature?
No
Links / references
/label feature