Uneven traffic loadbalancing among kas pods
Below are graphs over the last 30 days. As you can see, there is always a Pod that gets more requests, has more in-flight connections and hence has more goroutines and RAM usage. This is an issue because it makes RAM usage uneven and hence resource utilization suboptimal - we allocate X MB of RAM per pod, but only a single pod uses that much, other ones use much less and the reserved RAM is "wasted" (not used but also is not allocated to other pods). That single pod forces us to scale UP more than we would otherwise would.
sum by (pod) (
grpc_server_handled_total{app="kas",env="gprd"}
)
sum by (pod) (
grpc_server_requests_in_flight{app="kas",env="gprd"}
)
go_goroutines{app="kas",env="gprd"}
container_memory_working_set_bytes:labeled{env="gprd",type="kas"}