Integrated Load Testing
Load testing is a common and typically important stage in the CI process, ensuring changes to the app can scale before they are introduced into production.
Advanced deployment strategies like Canary and Blue/Green can mitigate the risk for deploying code that does not scale, however that is not a true alternative for a few reasons:
- You can't improve what you don't measure. Running repeatable and consistent performance tests lets you compare results between runs, to ensure performance doesn't slowly degrade over time.
- Canary and Blue/Green do minimize risk, but you are still essentially testing in production. Best practices would dictate that you use CI test what you can, to reduce the frequency and occurrence of rollbacks and errors in production.
There are a number of open source load testing tools, for example:
While we should eventually support the top few load testing tool output formats, we should test and select one for further integration with gitLab. With our integration, we could automatically configure and run the load tests as part of Auto DevOps.
Potential GitLab Solution
Automatically generate a load testing CI job against
- Allow customers to specify additional URL's to test (#3540)
- Utilize the existing review environment, or one customer specifies
- Archive and persist the performance results
- Allow for customers to specify their own load test scripts
- Display the results on the Merge Request page, similar to Performance, although we will indicate the source of the performance metrics (Load or Browser test)
The value of this being integrated with GitLab is to able to clearly show performance changes using a repeatable and consistent test, at the time of deciding whether to merge. This simplifies the developer workflow, by reducing the number of tools a reviewer needs to consult or look at. (Similar to CodeClimate)