[Meta] Load & Performance Testing Support
One of the major hurdles to embracing Continuous Delivery is ensuring that your company has the confidence to proceed with deployments in an automated fashion. As they mature on their journey to CD, they will continue to automate more and more of their SDLC and testing process. One of the important parts of testing a release, is ensuring that there have not been any performance regressions and it can handle expected load. This can take a few forms, such as a short term stress test or a longer term soak test.
Today GitLab does not have a great solution for solving this, and relies upon the customer to come up with their own via the CI/CD YML file. Prometheus can monitor and detect issues, but it cannot actually generate load. In a pre-production environment there typically is little to no load, and so monitoring results are not very valuable or actionable unless there is a serious regression.
GitLab should integrate with a performance testing tool to bridge this gap, helping to increase adoption of CI/CD.
There are a few potential open source products that are in this area, two examples are Gatling and Locust. We could ultimately support multiple, but should pick one for initial integration like we have done with Prometheus.
With some work on the GitLab side, we could make it easier to embrace these solutions, and enrich the process:
- Simplify where possible the process of writing and running these tests. [CE]
- Detect presence of test file within repository, automatically enable. Potentially add the lines required to CI YML. [CE]
- Other potential options based on tool.
- Present the test results within GitLab [CE] as job output [CE]
- Can be a simple 3 line chart: Requests/sec, Request Duration, Errors/sec.
- Aggregate and display results across multiple commits within a branch. Compare results against another branch. (master) [EE]
- Automatically create issues when: results over a predefined value, regression from prior results. [EE]
Depending on the tool, it may also be possible to output this data into Prometheus. For example there are a couple of instances where people have tried to add their own Prometheus exporter to Locust.io, although just comments no actual repo. This may be more effort than it is worth, though.
[Locust.io] is an open source tool written in Python. Tests are also written in Python. It is extensible and can support tests beyond HTTP. While it is being maintained, activity and responses to PR's is slow.
[Gatling.io] is also open source, but does have an enterprise edition similar to GitLab. These functionality appears to include features such as distributed testing to allow for higher load generation. It also integrates with a variety of CI tools already, with plugins available for Jenkins and Bamboo. Tests are written in their own proprietary language.
It is supported by a real company, with revenue coming from the enterprise edition sales. Development velocity appears higher than Locust.