Crossbrowser Testing - Overall Plan

Overview

We need to close out our test coverage gap for cross-browser and mobile browsers.

We do not want to run all of our tests in a 3rd party service. The majority of our test will be handled via our internal grid. We aim to have ~20% of our test automation bandwidth be tested on all browsers and on real devices if mobile browsers

We already have browserstack (maintained by the FE team) But this is a small account which also does not support SAML configuration.

Requirements

• Infrastructure

  • Handle all the flavor of browsers with many screen sizes
  • Handle mobile browser layouts
  • Handle the majority of mobile browsers that run on real devices.
  • Handle SSO auth for engineering

• Test coverage

  • At first we will run our existing smoke tests.
  • We will need to determine what are the browsers that we will support.
    • This can be a mix of industry numbers e.g. via http://gs.statcounter.com/browser-market-share
    • This also should be data driven taken from CS and also any in app analytics that we log.
  • We will add additional E2E visual validation tests in addition to the functional smoke tests.

Comparison

BrowserStack	SauceLabs

From a quick comparison it appears that the Saucelabs Enterprise plan seems to have more features.

Plan

Phase 1 - Initial gitlab-org&609

• Setup Saucelabs infrastructure with an enterprise plan that supports SSO. Saucelabs also supports mobile tests our of the box for the desktop plan.
• Framework support, we will need to ensure that the test runner can kick off test in Saucelabs. There has to be arguments that can specify run environments.
  • Flag to specify which env
  • Tag to enable cross-browser testing suite
• Coverage, determine first MVC of optimal browser coverage

Phase 2 - MVC of basic test running

• Identify cross-browser smoke test that needs to be run
  • Coordinate with CS and FE team as necessary
• Configure cross-browser test run for the current release process on staging
  • Fix tests as necessary, page object customization maybe needed.

Phase 3 - implementation at scale across engineering

• Integrate with Review Apps CI test pipeline
• Configure SAML SSO for engineering login.
• Address test coverage for dev stage, add more tests for cross-browser suite.

Phase 4 - Mobile coverage

• Page objects for mobile
• Functional tests for mobile