This epic collects issues related to an MVP-implementation of clustering and routing. The implementation will be based on ideas explored in the [poc/scaling-git](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/8812) branch, and discussed in the [Scaling Git: Routing and Clustering Design](https://docs.google.com/document/d/1RDcMw_fUI5uTlTcCmyqpeuC_E0BJCOwdyX3gghrjyy0/edit?tab=t.0) document. ### The goals for the MVP are: 1. Enable a fixed-size cluster of Gitaly nodes to be exposed as a single storage to Rails. 1. Allow for a fixed replication factor (i.e. the M value) across all repositories managed by the cluster. 1. Maintain cache stickiness via the routing layer, using a weighted rendezvous hashing scheme. 1. Perform routing through redirects instead of proxying. ### We are not tackling the following concerns for the MVP: 1. Cluster auto-scaling. The rationale is that we should be able achieve similar or better performance than Praefect, even with a fixed cluster size. Auto-scaling can be worked out later. 2. Setting per-repository replication factors. This can be addressed by consulting repository-specific metadata at a later stage. 3. Using health or resource usage metrics to influence routing. It is expected that a like-for-like Gitaly node using the MVCC backend will be able to handle the same amount of base traffic as a node using the files backend. 4. Cache-prewarming. Even if we implemented prewarming, the very first request to an unseen repository on a fresh cluster will incur significant hydration overhead anyway. 5. The use of Kubernetes-native constructs like a service mesh to perform routing and clustering, as described in https://gitlab.com/gitlab-org/gitaly/-/work_items/7259. We've still not received formal guidance on whether Kubernetes will be the sole deployment target, so we should remain environment-agnostic for now. 6. The deployment of a separate routing layer (ala Praefect). It's operationally simpler if the Gitaly nodes handle routing through middleware. We eliminate another point of failure, and gain the ability to scale all aspects of the cluster by changing the number of Gitaly nodes. 7. The deployment of a dedicated "Top K" cluster that exclusively handles traffic for busy repos. 8. The custom HELP protocol described in https://gitlab.com/gitlab-org/gitaly/-/work_items/7214. We can most definitely revisit better policies around distributing load at a future date. ### The components that need to be implemented for the MVP are: 1. [ ] A gossip protocol, so Gitaly nodes are aware of each other, the composition of the cluster, and the health of peer nodes. 1. [ ] A hashing scheme, so the router can determine the target(s) to direct the request to. See https://gitlab.com/gitlab-org/gitaly/-/work_items/7168 for an example. 1. [ ] Support for gRPC redirects. We don't want nodes to proxy requests like in the PoC or Praefect, so changes to the Gitaly client packages need to be made to support a custom redirect scheme. 1. [ ] A simple routing layer to intercept requests, invoke the hashing scheme, and direct the request as appropriate. 1. [ ] Support for client-side load balancing. We handle this in Praefect already by getting clients to [dial a DNS address](https://docs.gitlab.com/administration/gitaly/praefect/configure/#service-discovery), which removes the need for a dedicated load balancer. 1. [ ] A small tool to aid in debugging the cluster, since routing activity will be spread across multiple Gitaly nodes. Such a tool can simply ingest multiple log streams and build up the cluster state that way. ### This should give us the following end-state for the MVP implementation: 1. Gitaly nodes that can be configured to join a cluster based on knowledge of its peer addresses. 1. A Gitaly cluster that can be exposed to Rails as a new storage. An administrator can manually move repositories to this storage, or adjust the Rails' weights assigner to provision new repositories on the cluster. 1. Gitaly nodes that are stateless and ephemeral. Even though we can't auto-scale the cluster yet, we should have the ability to destroy and recreate nodes without loss of data or service interruption. 1. Clients that understand our custom gRPC redirect scheme and can successfully communicate with the appropriate Gitaly node for the requested repository.