How can we observe Praefect connection saturation?
Praefect, as a gRPC proxy/router, has the potential to become a bottleneck for passing messages around. If that happens, how would we detect it?
Situation without Praefect
In production we see that a single Gitaly server will have about 3000 open file descriptors. We believe (and need to verify) that this is mostly caused by having about 3000 incoming TCP connections. We believe (again, need to verify) that this number is roughly number of client processes
* number of services
, or in other words, for each gRPC service, each client will establish a TCP connection.
This means that currently, traffic going into a single Gitaly server gets spread out over 3000 TCP connections.
Situation with Praefect
I am now going to talk about "phase 1", i.e., Praefect is nothing more than a proxy for Gitaly. What used to be 1 Gitaly server now becomes: (GCP load balancer) -> 3x (Praefect) -> 1x (Gitaly).
From the point of view of Gitaly's clients, the GCP load balancer is a Gitaly server. So we should expect that load balancer to get about 3000 client connections. Those then get balanced across 3 Praefect nodes. What happens after that? Do we have only one TCP connection from each Praefect node to the Gitaly server in the back? If so, that means that traffic that used to be spread over 3000 connections now gets pushed to Gitaly over 3 connections.
That would be a big drop, and potentially, a bottleneck.
The question
If "traffic gets squeezed from 3000 TCP connections into 3" is a problem, how would we find out?