Integrated Load Testing MVC
Load testing is a common and typically important stage in the CI process, ensuring changes to the app can scale before they are introduced into production.
Advanced deployment strategies like Canary and Blue/Green can mitigate the risk for deploying code that does not scale, however that is not a true alternative for a few reasons:
- You can't improve what you don't measure. Running repeatable and consistent performance tests lets you compare results between runs, to ensure performance doesn't slowly degrade over time.
- Canary and Blue/Green do minimize risk, but you are still essentially testing in production. Best practices would dictate that you use CI test what you can, to reduce the frequency and occurrence of rollbacks and errors in production.
There are a number of open source load testing tools, for example:
While we should eventually support the top few load testing tool output formats, we should test and select one for further integration with gitLab. With our integration, we could automatically configure and run the load tests as part of Auto DevOps.
Our Quality group has begun to use Artillery.io to perform load testing against GitLab itself here: https://gitlab.com/gitlab-org/gitlab-ce/issues/57913.
Automatically generate a load testing CI job against
Allow customers to specify additional URL's to test (https://gitlab.com/gitlab-org/gitlab-ee/issues/3540)
Utilize the existing review environment, or one customer specifies
Archive and persist the performance results
Allow for customers to specify their own load test scripts
Display the results on the Merge Request page, similar to Performance, although we will indicate the source of the performance metrics (Load or Browser test)
The value of this being integrated with GitLab is to able to clearly show performance changes using a repeatable and consistent test, at the time of deciding whether to merge. This simplifies the developer workflow, by reducing the number of tools a reviewer needs to consult or look at. (Similar to CodeClimate)
In the outputted report type, we should create a generic interface that will be agnostic to the specific open source tool we select such that users can also "bring their own" testing suite to this feature.