Roadmap - E2E Test Parallelization
Overview
Gitlab QA tests are still running sequentially, I consider this the most basic form of running test. All tests should be able to run in parallel and they should be automated as such out of the box. This roadmap captures the steps to achieve full Parallelization running all the tests at the same time. The end goal is to have the whole test suite take as long as the longest test. Test reporting and logging would also have to be adjusted to account for isolated test processes
References
- Running all ui tests in 39 seconds https://aws.amazon.com/blogs/devops/ui-testing-at-scale-with-aws-lambda/
- Basic parallization in Ruby http://elementalselenium.com/tips/27-parallel
Requirements
- All end-to-end test suites will need to be able to run in parallel, no excuses.
- Test reporting, test data atomization, test isolation needs to be done out of the box.
- Test runner should be configurable, we should be able to adjust the level of parallelization e.g.
- Run all suites in parallel.
- Run all tests in parallel (every single test/specify a concurrency level)
- Test reporting needs to be constructed after the test run is finished.
- The test runner should output a test result object for every test, in the end we just merge the data and specify it to a reporter renderer.
- Test report builder should be a seperate process after the test finishes.
- We would want to deploy a dedicated selenium docker grid for each run.
Plan
Phase 1 - Basic tooling in place
- Decide on a parallel runner and framework
- Ensure that all the tests can be run in parallel
- Dedide on a html reporter implementation
- Achieve x2 to x4 parallel tests concurrency
Phase 2 - Integrate reporting and trial runs
- Build reporter with html reporting output to artifacts of tests
- Support for Increase parallel concurrency
- Ensure that reporting is accurage
- Trial run in CI
Phase 3 - Full concurrency
- Dedicated Selenium Grid Infrastructure
- Run all tests in parallel