Keep the RSpec reports when the test suite fails

What does this MR do and why?

When a spec fails, the test component uploads neither its test report nor its coverage report, so the pipeline's Tests tab and the merge request's test summary are empty in exactly the case they are there for.

The job runs bundle exec rspec, which writes rspec.xml and coverage/coverage.xml, and copies them in the next script entry to ${CI_JOB_ID}_junit.xml and ${CI_JOB_ID}_cobertura.xml, the files artifacts:reports names. When an example fails, rspec exits non-zero and the job stops at that entry (when script commands return an exit code other than zero, "the job fails and further commands do not execute"), so the copies never run and artifacts: when: always finds nothing to upload (lines from the job log):

WARNING: 2_junit.xml: no matching files. Ensure that the artifact path is relative to the working directory (/builds/root/ruby)
WARNING: 2_cobertura.xml: no matching files. Ensure that the artifact path is relative to the working directory (/builds/root/ruby)
ERROR: No files to upload

Until !6 (merged), artifacts:reports named the junit_report_path and coverage_report_path inputs themselves (rspec.xml and coverage/coverage.xml by default), which rspec writes before it exits, so with project_path: . a failing run still had its reports. !6 (merged) pointed artifacts:reports at copies made in a later script entry, and !12 (merged) made those copies run for every project_path, still after rspec.

This MR changes three lines in templates/test.yml:

• rspec_status=0 and bundle exec rspec || rspec_status=$? record rspec's exit status instead of stopping on it, as that page of the documentation shows. Setting it to 0 first, as the documentation's example does, also keeps a CI/CD variable named rspec_status from deciding the job's result.
• The block that adjusts and copies the reports runs as before and now ends with exit "$rspec_status", so the job still fails, with rspec's own exit code, when a test fails, and passes when none does.

I kept the copies in script rather than moving them to after_script: after_script is an input of this component, and it runs in a new shell that starts back in the build directory rather than in project_path.

References

• !6 (merged) and !12 (merged), which introduced the report copies and then made them run for every project_path.
• Ignore non-zero exit codes in the CI/CD YAML documentation.

Screenshots or screen recordings

The Tests tab of a pipeline that includes test with project_path: example_project and one failing example added, from main (before) and from this branch (after):

Before	After

How to set up and validate locally

1. Include test from this branch in a project and add a failing example, for instance spec/lib/failing_spec.rb:

   RSpec.describe 'A deliberately failing example' do
     it('fails') { expect(1 + 1).to eq(3) }
   end

2. Run the pipeline. The job still fails, its log shows Uploading artifacts as "junit" to coordinator... 201 Created, and the pipeline's Tests tab lists the failed example. On main the same job logs ERROR: No files to upload and the tab is empty.

3. Remove the failing example and run it again. The job passes and uploads both reports, as on main.

I ran these steps on a local GitLab CE 19.4.1 with GitLab Runner 19.4.1 (Docker executor, ruby:3.4), including test from the commit under test once per pipeline: with project_path: example_project, and in separate pipelines with project_path: . against a copy of example_project at the repository root. I did not add a failing example to example_pipelines: those child pipelines run with strategy: depend, so it would fail this project's own pipeline.

The pipelines on the local GitLab

Pipeline	project_path	Commit	Status	Tests tab	Report artifacts
main with a failing example	example_project	f00d971 + proof commit	failed	0 tests	none
this branch with a failing example	example_project	7b78e2f + proof commit	failed	3 tests, 1 failed, 1 passed, 1 skipped	junit, cobertura
this branch, suite as committed	example_project	7b78e2f + proof commit	passed	2 tests, 0 failed, 1 passed, 1 skipped	junit, cobertura
main with a failing example	.	f00d971 + proof commit	failed	0 tests	none
this branch with a failing example	.	7b78e2f + proof commit	failed	3 tests, 1 failed, 1 passed, 1 skipped	junit, cobertura
this branch, suite as committed	.	7b78e2f + proof commit	passed	2 tests, 0 failed, 1 passed, 1 skipped	junit, cobertura

Each proof commit replaced .gitlab-ci.yml with one include of $CI_SERVER_FQDN/$CI_PROJECT_PATH/test@$CI_COMMIT_SHA: job name rspec-example-project with project_path: example_project, or job name rspec-root with project_path: . and a copy of example_project at the repository root. The pipelines with a failing example add it to the suite the job runs. Each job's log shows $ cd example_project or $ cd ., matching its project_path. The skipped test is the pending example in example_spec.rb.

rspec-example-project on main with the failing example (lines from the job log):

$ cd example_project
$ bundle exec rspec
3 examples, 1 failure, 1 pending
WARNING: 2_junit.xml: no matching files. Ensure that the artifact path is relative to the working directory (/builds/root/ruby)
WARNING: 2_cobertura.xml: no matching files. Ensure that the artifact path is relative to the working directory (/builds/root/ruby)
ERROR: No files to upload
WARNING: 2_junit.xml: no matching files. Ensure that the artifact path is relative to the working directory (/builds/root/ruby)
ERROR: No files to upload
WARNING: 2_cobertura.xml: no matching files. Ensure that the artifact path is relative to the working directory (/builds/root/ruby)
ERROR: No files to upload
ERROR: Job failed: exit code 1

The same job on this branch (lines from the job log):

$ cd example_project
$ rspec_status=0
$ bundle exec rspec || rspec_status=$?
3 examples, 1 failure, 1 pending
$ if [ "example_project" != "." ]; then # collapsed multi-line command
1_junit.xml: found 1 matching artifact files and directories
1_cobertura.xml: found 1 matching artifact files and directories
Uploading artifacts as "archive" to coordinator... 201 Created
1_junit.xml: found 1 matching artifact files and directories
Uploading artifacts as "junit" to coordinator... 201 Created
1_cobertura.xml: found 1 matching artifact files and directories
Uploading artifacts as "cobertura" to coordinator... 201 Created
ERROR: Job failed: exit code 1

The failed example in the rspec-example-project suite, with the file path the project_path adjustment writes:

The job's script in ruby:3.4, in three runner modes

I rendered .ruby-base's before_script and the job's script from the templates with the inputs interpolated, and ran them in ruby:3.4 wrapped the way GitLab Runner's bash shell wraps a job script (errexit and pipefail, then eval in a subshell), with CI_JOB_ID=4242. I ran each case in the runner's default mode, with FF_USE_LEGACY_BASH_EVAL (: | eval) and with FF_ENABLE_BASH_EXIT_CODE_CHECK (an exit code check after each command), and the three modes gave the same result in every case:

project_path	Suite	main	This branch
.	one failing example	exit 1, no 4242_junit.xml, no 4242_cobertura.xml	exit 1, both files, the JUnit file with 3 tests and 1 failure in ./spec/lib/failing_spec.rb
example_project	one failing example	exit 1, no 4242_junit.xml, no 4242_cobertura.xml	exit 1, both files, the JUnit file with 3 tests and 1 failure in ./example_project/spec/lib/failing_spec.rb
.	as committed	exit 0, both files	exit 0, both files
example_project	as committed	exit 0, both files	exit 0, both files

Two more runs, both with project_path: example_project in the default mode:

• With --failure-exit-code 7 added to .rspec and the failing example, the job exits 7 on main and on this branch, so its exit status is still rspec's.
• With rspec_status=3 in the job's environment and the suite as committed, the job exits 0 on this branch.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.