Skip to content

Geo: Test Gitaly Cluster on a Geo secondary

Problem to solve

Gitaly now supports Gitaly Cluster for fault-tolerance. We should test if Gitaly Cluster works as expected for a Geo Secondary configured as multiple servers.

Proposal

  • Spin up a 5000 user reference architecture or another architecture that is sufficient to test multiple gitlab nodes. (NOTE: see Further details for system specs used)
  • Modify Gitaly nodes so that it supports Gitaly cluster
  • Run Geo end-to-end tests
  • Report if deployment is healthy
  • Run end-to-end Praefect/Gitaly automatic failover test (manually if required)
  • Open MR to add test details to Geo verification page

Further details

Each GitLab site was based on the 2,000 user reference architecture (excluding NFS) and adjusted to include a total of 3 Gitaly nodes, 3 Praefect nodes, and 1 Postgres-for-Praefect node.

  • Gitlab version: 13.3.0-pre(018ffad5)
  • PostgreSQL version: 11.7
  • Git version: 2.27.0
  • GitLab Shell version: 13.5.0
  • Gitlab Workhorse version: v.8.37.0
  • Ruby version 2.6.6.p146
  • Rails version: 6.0.3.1
  • GitLab API version: v4

Other Notes

  • Gitaly clusters were setup on fresh installs of GitLab, so there were no migrations of existing repositories to Praefect
  • Distributed reads feature was enabled

Documentation

This should be documented in the Geo testing section

Availability & Testing

  • Run Geo end-to-end tests
  • (Aug 17 update) Trigger an automatic failover on primary Gitaly cluster, check that failover is complete, and then check git push operations before and after failover. Then trigger failover on secondary Gitaly cluster (and test git pushes before and after failover).

What does success look like, and how can we measure that?

  • Geo secondary functioning as expected using Gitaly Cluster

What is the type of buyer?

  • Premium

Is this a cross-stage feature?

  • Yes. Enablement. Geo. Source Code.

Links / references

Edited by Jennifer Louie