Create a benchmarking tool (load testing) for GitLab
The (PostgreSQL) database team needs to have a performance benchmarking tool, and a baseline performance measurement, to test configuration and architecture changes to the database and validate them, from a performance perspective, before deploying to production.
We have now several challenges for changes in production that should not be deployed to production without proper performance benchmarking (among other validations) like:
- Evaluate a different connection pooler (see https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7776)
- Tune shared_buffers and other memory related parameters after the upcoming hardware upgrade (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7778).
- Consider using
huge pages
for PostgreSQL (see https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7819).
There have been previous attempts to do this at the database level. In particular, a well-known method is to set log_min_duration_satatement to 0
and log all the queries that hit the database. Then, with a tool like pgreplay, a workload is created and can be replayed anywhere, at any pace. This helps understand the performance implications of any production change. However, these mechanism doesn't work on GitLab due to the high rate of incoming queries, which can overwhelm the logging system and harm the production services.
We're left as of now with application-level performance benchmarking. I expect a framework that would exercise HTTP calls following patterns similar to the ones that can be observed in production, running against a staging cluster similar in data and characteristics to the ones in production, and thus generating a realistic workload. If well designed, it should hit the database in a similarly realistic way, comparably to the traffic and load that the current production database receives.
I guess such a framework already exists at GitLab, and then this issue is created just to let the database team know where it is and how to use it. If not, it becomes the proposal to create such a framework. As an example, there are some tools to easy this task, like Locust, but there might be more appropriate tools for GitLab.