Testbed for on-prem performance testing
Background
Quality team is working on closing out performance test gaps for on-prem customers. Tracked in gitlab-org&805 (closed). Production is no longer a realistic test instance for on-prem customers due to the abundance of computing power. Review apps are also not suitable because most of our on-prem uses omnibus installer on a native OS (VMs) and not K8s.
Recent customer RCAs (GitLab only)
Requirements
This testbed is supposed to help supplement the gap between review apps and staging. The intent is to have this be around for a while since performance tests would be running on this regularly. We would like to start with one test bed for now.
Testbed overview
a single VM omnibus GitLab EE deployment, deployed from master, which has at least one shard mounted on NFS will be sufficient.
I think we'll want PostgreSQL and Redis to be on separate nodes because N+1 queries may not manifest as performance issues if the queries are done locally. Basically, I think the minimum setup should be:
- 2 x GitLab worker machines
- 1 x NFS/Gitaly server
- 1 x Redis node
- 1 x PostgreSQL node
FYI https://gitlab.com/stanhu/ansible-gitlab-geo#primary-cluster provisions these machines via Ansible on Google Cloud.
Instrumentation We would also like to add instrumentation with Sitespeed and build on top of the tooling that the frontend team has already done. The goal is to have test runs execute against this regularly and start spotting performance degradation before it hits our on-prem customers
Network access We will be using a blend of performance tests from GitLab QA and running tests from outside to simulate real usage. Hence, network and API access is needed. As also ability to run a bombardment of test from external CI runners.
Regular deployments
Given that we aim to complete our test runs within 1-2 days, the deployment to the test bed would be frequent and on the latest master.
@glopezfernandez FYI this is as what we discussed earlier.
/cc @at.ramya