Investigate gRPC compression in Gitaly
Run an experiment to see what happens to latency, cpu usage and network bandwidth if we turn on (some level of) gRPC compression.
- Network costs of Gitaly are significant: https://10az.online.tableau.com/#/site/gitlab/views/FinOps-Unit-economics-Gitaly/UE-Gitaly-Requests?:iid=1
- gRPC does not use any compressors by default: https://github.com/grpc/grpc-go/blob/master/Documentation/encoding.md#compressors-compression-and-decompression
- we (very likely) don't have compression on
- CPU is not a concern in availability or cost at the moment
- we may not gain much, quoting @pks-gitlab on slack :
I don't think we do. I think this was discussed once in the past, and we decided against it. The reason is mostly that git-receive-pack(1) and git-upload-pack(1) and their respective endpoints are by far the single-most-biggest RPCs regarding the number of bytes transferred. And those RPCs use the packfile format, which is compressed using zlib already. Thus, recompressing it won't really work and extra compression on top is a wasted effort.
Still, a low % gain would still add up to a lot of money.
NB: compression is assymetric, and can be chosen (and changed) at runtime.
cf https://github.com/grpc/grpc/blob/master/doc/compression_cookbook.md