Reuse Gitaly connections and sidechannel
When gitlab-sshd has been introduced we've started running our own SSH server. In this case we're able to cache and reuse Gitaly connections and Registry. It helps to reduce memory usage.
The changes are inspired by https://gitlab.com/gitlab-org/gitlab/-/blob/master/workhorse/internal/gitaly/gitaly.go
The number of cache entries depends on the number of Gitaly servers that we have. The cache key consists of:
- Service name (git-upload-pack|git-receive-pack|git-upload-archive). Cardinality: 3
- Gitaly server address. Cardinality: N
- Token. Usually belongs to Gitaly server
- Whether sidechannel is used. Cardinality: 2
The number of entries in cache is going to be 6 * N, unless a token is changed for a Gitaly server. The connections stay open to be reused and will be terminated on server termination.
Prometheus metrics are added to track the number of opened Gitaly connections:
gitlab_shell_gitaly_connections_total{status="ok"} 2On my local tests, multiple simultaneous git clone operations with sidechannel enabled are using up to 450MiB of memory. When the connections are reused, the memory usage seems to be capped and ~300MiB and doesn't grow further (hope it behaves the same on production):
- Left spikes - Before the MR
- Right spikes - After the MR
Related to #520 (closed)