Make Junit error -> screenshot mapping available to Front End
Problem to solve
#6061 will display screenshots stored as artifacts that correspond to failures in Junit but that mapping isn't consistent in the error data. Create and provide a mapping.
Intended users
This is for internal GitLab use to support #6061.
Further details
This supports issue #6061 by providing a link to an artifact captured in a test to be pulled into the front end.
Some use cases to consider:
- File is available
- No file is available / pass an error the front end can deal with
- This could be a file is missing (most likely) and the front end should get that info
- This may be invalid format? (we may make the front end handle that)
Proposal
Current proposal
1. Change our screenshots names on test failures
We need to provide a 1:1 mapping between our RSpec
failures and our JUnit
xml in order to figure out where to fetch this screenshot. This could be done by leveraging the full_description of an RSpec
object which is accessible in JUnit
with the name attributes.
This means we will change the way we persist screenshots from this convention to this one:
Before | After |
---|---|
../browser_ui/1_manage/group/groups_permission_checks_spec.rb_2020-02-28-17-25-22.691.png |
../verify_pipeline_creation_and_processing_users_creates_a_pipeline_which_gets_processed.html |
This can be done by changing how we persist screenshot with Capybara
Capybara::Screenshot.append_timestamp = false
Capybara::Screenshot.register_filename_prefix_formatter(:rspec) do |example|
::File.join(QA::Runtime::Namespace.name, example.full_description.downcase.parameterize(separator: "_"))
end
With our current RSpec
setting each failing test gets assigned a screenshot metadata to it which makes it easier to understand where the screenshot lives.
Next challenge here would be to "persist" this data somewhere. Which bring me to my next point.
2. CustomFormatter
We have 2 options here:
We contribute to the current formatter we are using today- We decided to add the screenshot metadata generated by
Capybara
to our rspec failure:
if example.metadata[:screenshot]
screenshot = example.metadata[:screenshot][:image] || example.metadata[:screenshot][:html]
example.metadata[:stdout] = %{[[ATTACHMENT|#{screenshot}]]}
end
3. Parse JUnit Report
Now that this information lives in a JUnit XML report
file we would need to parse it and store in order to be able to send it to the front end.
JUnit XML
report containing a system-out
tag with the ATTACHMENT
which will can now be parsed by GitLab. This data is now available on a TestCase object.
<system-out>
[[ATTACHMENT|/Users/goldenson/Code/gitlab-development-kit/gitlab/qa/tmp/qa-test-2020-03-27-13-39-00-76dfdbbdfe527962/verify_pipeline_creation_and_processing_users_creates_a_pipeline_which_gets_processed.png]]
</system-out>
4. Expose screenshot to the front end
-
A new attribute has been added to our
TestCase API
introducing an attachment_url which contains the full path to the failed screenshot. -
We will expose a new
scope=with_attachment
attribute on our test_report endpoint. This way the front end will be able to display failing screenshots for a given pipeline.
Test report without attachment | Test report with attachment |
---|---|
Permissions and Security
This feature is behind the junit_pipeline_screenshots_view
feature flag (attachment_url) and only for internal purpose at the moment.
Documentation
- N/A if this is internal.
- Update API docs is this is available.
Availability & Testing
@zeffmorgan can you take a look here at your convenience?
What does success look like, and how can we measure that?
If successful this will support implementation of #6061.
What is the type of buyer?
N/A