Visual Regression Testing - Overall Plan
Overview
In addition to functional validation, we should also cover our bases in visual regression validation.
We have seen a number of visual regressions snug through. E.g.
- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21697#note_101775906
- https://gitlab.slack.com/archives/C3MAZRM8W/p1536750760000100
We already have basic visual regressions in place at the component levels. But these are more like unit tests for visual regressions. We still need a visual regression layer in the E2E layer that verifies the page as a whole when all the components are used. Visual integration tests are great, but we will want checks in other layers in addition to the lower level tests.
See visual regression testing pyramid
Requirements
-
Flexibility and ease of use
- The visual regression tool should be flexible and can mask certain areas and able to ignore data and verify layouts
- Since the data will be on staging and even production we need to account for wildcard data. e.g. We only want to validate that the page looks correct.
- Easy baseline management, if possible we will want to use the same baseline for staging and production so change management happens just once.
-
Test coverage
- At first we will convert some of our existing smoke tests to a visual validation test.
- Determine the screens that are the most important. E.g.
- A list of issues with multiple labels
- A MR with discussions and labels
- An issue with discussions and labels
- CI/CD widgetns
- Iterate
Comparison
Applitools | Features |
---|---|
https://applitools.com/ ! |
Plan
Phase 1 - Initial
- Identify visual regression engine that we will use
- Since we need real browsers, this should work well with our cross browser tooling and infrastructure
- Framework support, easy to convert our exising capybara tests over and add visual snapshots.
Phase 2 - MVC of basic test running
- Identify cross browser coverage
- Run first iteration of tests and create the baseline
- Configure this to run on staging as part of every release process.
- Reporting mechanism since this is a new type of tests.
Phase 3 - Increase coverage and bake into the release process
- Integrate with release process pipeline.
- Visual diff failure should block releases
- Address test coverage for dev stage, add more tests for cross-browser suite.
Phase 4 - Mobile coverage
- Integrate with mobile cross browser testing
- Mobile browser coverage
/cc @gl-quality
Edited by Ramya Authappan