Crossbrowser Testing - Overall Plan
Overview
We need to close out our test coverage gap for cross-browser and mobile browsers.
We do not want to run all of our tests in a 3rd party service. The majority of our test will be handled via our internal grid. We aim to have ~20% of our test automation bandwidth be tested on all browsers and on real devices if mobile browsers
We already have browserstack (maintained by the FE team) But this is a small account which also does not support SAML configuration.
Requirements
-
Infrastructure
- Handle all the flavor of browsers with many screen sizes
- Handle the majority of mobile browsers that run on real devices.
- Handle SSO auth for engineering
-
Test coverage
- At first we will run our existing smoke tests.
- We will need to determine what are the browsers that we will support.
- This can be a mix of industry numbers e.g. via http://gs.statcounter.com/browser-market-share
- This also should be data driven taken from CS and also any in app analytics that we log.
- We will add additional E2E visual validation tests in addition to the functional smoke tests.
Comparison
| BrowserStack | SauceLabs |
|---|---|
 |
 |
From a quick comparison it appears that the Saucelabs Enterprise plan seems to have more features.
Plan
Phase 1 - Initial gitlab-org&609
- Setup Saucelabs infrastructure with an enterprise plan that supports SSO. Saucelabs also supports mobile tests our of the box for the desktop plan.
- Framework support, we will need to ensure that the test runner can kick off test in Saucelabs. There has to be arguments that can specify run environments.
- Flag to specify which env
- Tag to enable cross-browser testing suite
- Coverage, determine first MVC of optimal browser coverage
Phase 2 - MVC of basic test running
- Identify cross-browser smoke test that needs to be run
- Coordinate with CS and FE team as necessary
- Configure cross-browser test run for the current release process on staging
- Fix tests as necessary, page object customization maybe needed.
Phase 3 - implementation at scale across engineering
- Integrate with Review Apps CI test pipeline
- Configure SAML SSO for engineering login.
- Address test coverage for dev stage, add more tests for cross-browser suite.
Phase 4 - Mobile coverage
- Page objects for mobile
- Functional tests for mobile