gRPC: Sentry reports with 'connection closed' error from time to time.
From time to time Sentry report with next error
*status.statusError: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection closed
According to the gRPC doc this could be because of absent activity throw the connection or wring configuration.
Yes, we have configured keepalive option, but looks like this is still could be an issue because of https://github.com/grpc/grpc-go/issues/2638 and currently used version is v1.24.0
.
Another potential problem is that we have RegisterNode uses both secure WithPerRPCCredentials
explicitly and WithInsecure
implicitly added by gitlab.com/gitlab-org/gitaly/client.Dial
for setting up connection to the Gitaly node.