Geo: Test Gitaly Cluster on a Geo secondary
Problem to solve
Gitaly now supports Gitaly Cluster for fault-tolerance. We should test if Gitaly Cluster works as expected for a Geo Secondary configured as multiple servers.
Proposal
-
Spin up a 5000 user reference architecture or anotherarchitecture that is sufficient to test multiple gitlab nodes. (NOTE: see Further details for system specs used) -
Modify Gitaly nodes so that it supports Gitaly cluster -
Run Geo end-to-end tests -
Report if deployment is healthy -
Run end-to-end Praefect/Gitaly automatic failover test (manually if required) -
Open MR to add test details to Geo verification page
Further details
Each GitLab site was based on the 2,000 user reference architecture (excluding NFS) and adjusted to include a total of 3 Gitaly nodes, 3 Praefect nodes, and 1 Postgres-for-Praefect node.
- Gitlab version: 13.3.0-pre(018ffad5)
- PostgreSQL version: 11.7
- Git version: 2.27.0
- GitLab Shell version: 13.5.0
- Gitlab Workhorse version: v.8.37.0
- Ruby version 2.6.6.p146
- Rails version: 6.0.3.1
- GitLab API version: v4
Other Notes
- Gitaly clusters were setup on fresh installs of GitLab, so there were no migrations of existing repositories to Praefect
- Distributed reads feature was enabled
Documentation
This should be documented in the Geo testing section
Availability & Testing
- Run Geo end-to-end tests
- (Aug 17 update) Trigger an automatic failover on primary Gitaly cluster, check that failover is complete, and then check git push operations before and after failover. Then trigger failover on secondary Gitaly cluster (and test git pushes before and after failover).
What does success look like, and how can we measure that?
- Geo secondary functioning as expected using Gitaly Cluster
What is the type of buyer?
- Premium
Is this a cross-stage feature?
- Yes. Enablement. Geo. Source Code.
Links / references
Edited by Jennifer Louie