Horizontal scaling of a single shard MVC

Problem to solve

When hundreds of developers are working on a mono repo (perhaps 5GB), it is possible to saturate a single server with reads and writes. CI servers add to this load. There is no way to horizontally scale reads without NFS today.

Further details

The current workaround for this is:

• install a load balancer between GitLab and a Gitaly storage
• the load balancer should distribute reads and writes across all many Gitaly nodes
• Gitaly nodes are kept in sync by a shared NFS volume that is mounted to each Gitaly nodes

This relies on NFS to keep all nodes completely consistent.

Proposal

When using Praefect for Gitaly HA, each repository is replicated some number of times. Only the primary node services reads.

Using the Praefect tracking database:

• randomly distributes reads to any node where the replication queue is empty for that repository.

Links / references