Add RSpec stdout metadata (!27128) · Merge requests · GitLab.org / GitLab

Part of #202114 (closed)

What does this MR do?

Following up !26788 (merged) where we modified our capybara screenshots file names on test failures.

This MR adds a stdout metadata to an RSpec object.

Why are we adding this?

Today's problem

It's not possible to link a screenshot to a test failure. If a failure happens we need to inspect the job artifacts manually which takes time and it's a hard task for us internally.

Solution

We want to make it simple for the user to display a screenshot associated to a test failure. This way we can show a failing screenshot on a pipeline without browsing all the job artifacts manually.

Check the design for this feature.

By creating a new stdout metadata on an RSpec object it will allow us to get this data injected properly in our JUnit XML reports.

Below an example of what this change will generate on our JUnit XML reports.

<system-out>[[ATTACHMENT|/Users/goldenson/Code/gitlab-development-kit/gitlab/qa/tmp/qa-test-2020-03-12-14-50-08-f2e50c63c279fdad/verify_pipeline_creation_and_processing_users_creates_a_pipeline_which_gets_processed.png]]</system-out>

This solution was proposed by the the creator of the RspecJunitFormatter gem and seems to perfectly fit what we are trying to achieve here.

Screenshots

If you want to try this feature, follow these steps:

git checkout mo-add-rspec-stdout-metadata
cd qa && bundle install
bundle exec bin/qa Test::Instance::All http://localhost:3000 -- qa/specs/features/browser_ui/4_verify/pipeline/create_and_process_pipeline_spec.rb --format RspecJunitFormatter --out rspec.xml

This will generate the following XML including a new system-out tag containing our attachment:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="rspec" tests="1" skipped="0" failures="1" errors="0" time="13.110646" timestamp="2020-03-12T14:50:08-04:00" hostname="Maximes-MacBook-Pro.local">
  <properties>
    <property name="seed" value="40622"/>
  </properties>
  <testcase classname="qa.specs.features.browser_ui.4_verify.pipeline.create_and_process_pipeline_spec" name="Verify Pipeline creation and processing users creates a pipeline which gets processed" file="./qa/specs/features/browser_ui/4_verify/pipeline/create_and_process_pipeline_spec.rb" time="13.106098">
    <failure message="Command `docker pull gitlab/gitlab-runner:alpine` failed!" type="QA::Service::Shellout::CommandError">Failure/Error:
  Resource::Runner.fabricate! do |runner|
    runner.project = project
    runner.name = executor
    runner.tags = [executor]
  end

QA::Service::Shellout::CommandError:
  Command `docker pull gitlab/gitlab-runner:alpine` failed!
./qa/service/shellout.rb:25:in `block in shell'
./qa/service/shellout.rb:19:in `shell'
./qa/service/docker_run/base.rb:22:in `pull'
./qa/resource/runner.rb:33:in `block in fabricate_via_api!'
./qa/resource/runner.rb:32:in `tap'
./qa/resource/runner.rb:32:in `fabricate_via_api!'
./qa/resource/base.rb:46:in `block (2 levels) in fabricate_via_api!'
./qa/resource/base.rb:126:in `log_fabrication'
./qa/resource/base.rb:46:in `block in fabricate_via_api!'
./qa/resource/base.rb:118:in `do_fabricate!'
./qa/resource/base.rb:45:in `fabricate_via_api!'
./qa/resource/base.rb:19:in `fabricate!'
./qa/specs/features/browser_ui/4_verify/pipeline/create_and_process_pipeline_spec.rb:16:in `block (3 levels) in &lt;module:QA&gt;'
./qa/runtime/browser.rb:55:in `block (2 levels) in configure!'
./qa/runtime/browser.rb:55:in `block (2 levels) in configure!'
./qa/specs/runner.rb:69:in `perform'
./qa/scenario/template.rb:10:in `block in perform'
./qa/scenario/template.rb:8:in `tap'
./qa/scenario/template.rb:8:in `perform'
./qa/scenario/template.rb:33:in `perform'
./qa/scenario/template.rb:10:in `block in perform'
./qa/scenario/template.rb:8:in `tap'
./qa/scenario/template.rb:8:in `perform'
./qa/scenario/bootable.rb:28:in `launch!'</failure>
    <system-out>[[ATTACHMENT|/Users/goldenson/Code/gitlab-development-kit/gitlab/qa/tmp/qa-test-2020-03-12-14-50-08-f2e50c63c279fdad/verify_pipeline_creation_and_processing_users_creates_a_pipeline_which_gets_processed.png]]</system-out>
  </testcase>
</testsuite>

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
Tested in all supported browsers
Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

Label as security and @ mention @gitlab-com/gl-security/appsec
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
Security reports checked/validated by a reviewer from the AppSec team

Add RSpec stdout metadata