ReplicateRepository timing out when used to move repositories
During the demo, we observed that moving www-gitlab-com failed because ReplicateRepository errored with the following message
Praefect:
{"correlation_id":"khGAikStsT9","error":"rpc error: code = Unavailable desc = transport is closing","grpc.code":"Unavailable","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.method":"ReplicateRepository","grpc.request.deadline":"2020-04-29T06:16:24Z","grpc.request.fullMethod":"/gitaly.RepositoryService/ReplicateRepository","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2020-04-29T00:16:24Z","grpc.time_ms":160014.56,"level":"warning","msg":"finished streaming call with code Unavailable","peer.address":"10.150.0.43:55610","pid":18317,"sentry.skip":{},"span.kind":"server","system":"grpc","time":"2020-04-29T00:19:04.939Z"}Gitaly:
{"correlation_id":"khGAikStsT9","error":"rpc error: code = Unavailable desc = transport is closing","grpc.code":"Unavailable","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.method":"ReplicateRepository","grpc.request.deadline":"2020-04-29T06:16:24Z","grpc.request.fullMethod":"/gitaly.RepositoryService/ReplicateRepository","grpc.request.glProjectPath":"root/www-gitlab-com","grpc.request.glRepository":"project-1","grpc.request.repoPath":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","grpc.request.repoStorage":"gitaly-1","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2020-04-29T00:16:24Z","grpc.time_ms":160007.03,"level":"warning","msg":"finished unary call with code Unavailable","peer.address":"10.150.0.48:60174","pid":18409,"span.kind":"server","system":"grpc","time":"2020-04-29T00:19:04.934Z"}Looks like this is also happening on prod: https://sentry.gitlab.net/gitlab/gitaly-production/issues/1211366/?query=transaction%3A%22RepositoryService%3A%3AReplicateRepository%22