Make Junit error -> screenshot mapping available to Front End

Problem to solve

#6061 will display screenshots stored as artifacts that correspond to failures in Junit but that mapping isn't consistent in the error data. Create and provide a mapping.

Intended users

This is for internal GitLab use to support #6061.

Further details

This supports issue #6061 by providing a link to an artifact captured in a test to be pulled into the front end.

Some use cases to consider:

• File is available
• No file is available / pass an error the front end can deal with
  • This could be a file is missing (most likely) and the front end should get that info
  • This may be invalid format? (we may make the front end handle that)

Proposal

Current proposal

1. Change our screenshots names on test failures

We need to provide a 1:1 mapping between our RSpec failures and our JUnit xml in order to figure out where to fetch this screenshot. This could be done by leveraging the full_description of an RSpec object which is accessible in JUnit with the name attributes.

This means we will change the way we persist screenshots from this convention to this one:

Before	After
../browser_ui/1_manage/group/groups_permission_checks_spec.rb_2020-02-28-17-25-22.691.png	../verify_pipeline_creation_and_processing_users_creates_a_pipeline_which_gets_processed.html

This can be done by changing how we persist screenshot with Capybara

Capybara::Screenshot.append_timestamp = false
Capybara::Screenshot.register_filename_prefix_formatter(:rspec) do |example|
  ::File.join(QA::Runtime::Namespace.name, example.full_description.downcase.parameterize(separator: "_"))
end

With our current RSpec setting each failing test gets assigned a screenshot metadata to it which makes it easier to understand where the screenshot lives.

Next challenge here would be to "persist" this data somewhere. Which bring me to my next point.

2. CustomFormatter

We have 2 options here:

• We contribute to the current formatter we are using today
• We decided to add the screenshot metadata generated by Capybara to our rspec failure:

if example.metadata[:screenshot]
  screenshot = example.metadata[:screenshot][:image] || example.metadata[:screenshot][:html]
  example.metadata[:stdout] = %{[[ATTACHMENT|#{screenshot}]]}
end

3. Parse JUnit Report

Now that this information lives in a JUnit XML report file we would need to parse it and store in order to be able to send it to the front end.

JUnit XML report containing a system-out tag with the ATTACHMENT which will can now be parsed by GitLab. This data is now available on a TestCase object.

<system-out>
  [[ATTACHMENT|/Users/goldenson/Code/gitlab-development-kit/gitlab/qa/tmp/qa-test-2020-03-27-13-39-00-76dfdbbdfe527962/verify_pipeline_creation_and_processing_users_creates_a_pipeline_which_gets_processed.png]]
</system-out>

4. Expose screenshot to the front end

1. A new attribute has been added to our TestCase API introducing an attachment_url which contains the full path to the failed screenshot.

2. We will expose a new scope=with_attachment attribute on our test_report endpoint. This way the front end will be able to display failing screenshots for a given pipeline.

Test report without attachment	Test report with attachment

Permissions and Security

This feature is behind the junit_pipeline_screenshots_view feature flag (attachment_url) and only for internal purpose at the moment.

Documentation

• N/A if this is internal.
• Update API docs is this is available.

Availability & Testing

@zeffmorgan can you take a look here at your convenience?

What does success look like, and how can we measure that?

If successful this will support implementation of #6061.

What is the type of buyer?

N/A

Links / references