Random 500 errors when cloning mirrored repository
User is seeing random 500 errors when cloning a repository that's mirrored in Bitbucket. Running GitLab 11.10.4 with gitaly running on its own server.
2019-05-22_14:07:58.18910 gitlab.domain.com x.x.x.x:0 - -
[2019/05/22:10:07:58 -0400] "GET
/group/project-client/-/jobs/123/trace.json?state= HTTP/1.1" 200 46 "
https://gitlab.
domain.com/group/project-client/-/jobs/123" "Mozilla/5.0 (Windows NT
10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/74.0.3729.108 Safari/537.36" 0.066
2019-05-22_14:07:58.63257 time="2019-05-22T10:07:58-04:00" level=error
msg=error correlation_id=eLO0EPSNv67 error="handleGetInfoRefs:
GetInfoRefsHandler: copy Gitaly response: rpc error:
code = Unavailable desc = transport is closing" method=GET
uri="/group/project-client.git/info/refs?service=git-upload-pack"
2019-05-22_14:07:58.63261 gitlab.domain.com x.x.x.x:0 - -
[2019/05/22:10:07:58 -0400] "GET
/group/project-client.git/info/refs?service=git-upload-pack HTTP/1.1" 500
22 "" "git/2.9.3" 0.077
Gitaly logs do not show a corresponding error:
# Different time and day but getting the same error in workhorse
2019-05-21_19:12:18.77613 time="2019-05-21T19:12:18Z" level=info msg="finished
streaming call with code OK"
correlation_id=ccedf938-4c7b-404b-a9da-89e959597626 grpc.code=OK
grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-sidekiq
grpc.method=FindAllTags grpc.request.deadline="2019-05-21T19:12:48Z"
grpc.request.fullMethod=/gitaly.RefService/FindAllTags
grpc.request.glProjectPath=group/project-client
grpc.request.glRepository=project-202
grpc.request.repoPath=group/project-client.git
grpc.request.repoStorage=default grpc.request.topLevelGroup=group
grpc.service=gitaly.RefService grpc.start_time="2019-05-21T19:12:18Z"
grpc.time_ms=6.944 peer.address="x.x.x.x:52870" span.kind=server
system=grpc
2019-05-21_19:12:18.78646 time="2019-05-21T19:12:18Z" level=info msg="finished
unary call with code OK" correlation_id=ccedf938-4c7b-404b-a9da-89e959597626
grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-sidekiq
grpc.method=AddRemote grpc.request.deadline="2019-05-21T19:12:28Z"
grpc.request.fullMethod=/gitaly.RemoteService/AddRemote
grpc.request.glProjectPath=group/project-client
grpc.request.glRepository=project-202
grpc.request.repoPath=group/project-client.git
grpc.request.repoStorage=default grpc.request.topLevelGroup=group
grpc.service=gitaly.RemoteService grpc.start_time="2019-05-21T19:12:18Z"
grpc.time_ms=2.11 peer.address="x.x.x.x:52870" span.kind=server
system=grpc
2019-05-21_19:12:18.78984 time="2019-05-21T19:12:18Z" level=info msg="I,
[2019-05-21T15:12:18.789191 #24048] INFO -- : Fetching remote upstream for
repository /var/opt/gitlab/git-data/repositories/group/project-client.git."
supervisor.args="[bundle exec bin/ruby-cd /var/opt/gitlab/gitaly
/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby
24025 /tmp/gitaly-ruby446221323/socket.1]"
supervisor.name=gitaly-ruby.1
2019-05-21_19:12:18.92385 time="2019-05-21T19:12:18Z" level=info msg="finished
unary call with code OK" correlation_id=ccedf938-4c7b-404b-a9da-89e959597626
grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-sidekiq
grpc.method=FetchRemote
grpc.request.fullMethod=/gitaly.RepositoryService/FetchRemote
grpc.request.glProjectPath=group/project-client
grpc.request.glRepository=project-202
grpc.request.repoPath=group/project-client.git
grpc.request.repoStorage=default grpc.request.topLevelGroup=group
grpc.service=gitaly.RepositoryService grpc.start_time="2019-05-21T19:12:18Z"
grpc.time_ms=135.442 peer.address="x.x.x.x:52870" span.kind=server
system=grpc
2019-05-21_19:12:18.93738 time="2019-05-21T19:12:18Z" level=info msg="finished
streaming call with code OK"
correlation_id=ccedf938-4c7b-404b-a9da-89e959597626 grpc.code=OK
grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-sidekiq
grpc.method=FindAllTags grpc.request.deadline="2019-05-21T19:12:49Z"
grpc.request.fullMethod=/gitaly.RefService/FindAllTags
grpc.request.glProjectPath=group/project-client
grpc.request.glRepository=project-202
grpc.request.repoPath=group/project-client.git
grpc.request.repoStorage=default grpc.request.topLevelGroup=group
grpc.service=gitaly.RefService grpc.start_time="2019-05-21T19:12:18Z"
grpc.time_ms=8.721 peer.address="x.x.x.x:52870" span.kind=server
system=grpc
2019-05-21_19:12:18.94853 time="2019-05-21T19:12:18Z" level=info msg="finished
streaming call with code OK"
correlation_id=ccedf938-4c7b-404b-a9da-89e959597626 grpc.code=OK
grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-sidekiq
grpc.method=FindLocalBranches grpc.request.deadline="2019-05-21T19:12:29Z"
grpc.request.fullMethod=/gitaly.RefService/FindLocalBranches
grpc.request.glProjectPath=group/project-client
grpc.request.glRepository=project-202
grpc.request.repoPath=group/project-client.git
grpc.request.repoStorage=default grpc.request.topLevelGroup=group
grpc.service=gitaly.RefService grpc.start_time="2019-05-21T19:12:18Z"
grpc.time_ms=8.546 peer.address="x.x.x.x:52870" span.kind=server
system=grpc
2019-05-21_19:12:18.95944 time="2019-05-21T19:12:18Z" level=info msg="finished
streaming call with code OK"
correlation_id=ccedf938-4c7b-404b-a9da-89e959597626 grpc.code=OK
grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-sidekiq
grpc.method=FindAllRemoteBranches
grpc.request.fullMethod=/gitaly.RefService/FindAllRemoteBranches
grpc.request.glProjectPath=group/project-client
grpc.request.glRepository=project-202
grpc.request.repoPath=group/project-client.git
grpc.request.repoStorage=default grpc.request.topLevelGroup=group
grpc.service=gitaly.RefService grpc.start_time="2019-05-21T19:12:18Z"
grpc.time_ms=8.828 peer.address="x.x.x.x:52870" span.kind=server
system=grpc
2019-05-21_19:12:18.98427 time="2019-05-21T19:12:18Z" level=info msg="finished
unary call with code OK" correlation_id=ccedf938-4c7b-404b-a9da-89e959597626
grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-sidekiq
grpc.method=DeleteRefs grpc.request.fullMethod=/gitaly.RefService/DeleteRefs
grpc.request.glProjectPath=group/project-client
grpc.request.glRepository=project-202
grpc.request.repoPath=group/project-client.git
grpc.request.repoStorage=default grpc.request.topLevelGroup=group
grpc.service=gitaly.RefService grpc.start_time="2019-05-21T19:12:18Z"
grpc.time_ms=7.799 peer.address="x.x.x.x:52870" span.kind=server
system=grpc
2019-05-21_19:12:19.06120 time="2019-05-21T19:12:19Z" level=info msg="finished
unary call with code OK" correlation_id=ccedf938-4c7b-404b-a9da-89e959597626
grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-sidekiq
grpc.method=GarbageCollect
grpc.request.fullMethod=/gitaly.RepositoryService/GarbageCollect
grpc.request.glProjectPath=group/project-client
grpc.request.glRepository=project-202
grpc.request.repoPath=group/project-client.git
grpc.request.repoStorage=default grpc.request.topLevelGroup=group
grpc.service=gitaly.RepositoryService grpc.start_time="2019-05-21T19:12:19Z"
grpc.time_ms=57.388 peer.address="10.73.241.140:51252" span.kind=server
system=grpc
2019-05-21_19:12:19.07009 time="2019-05-21T19:12:19Z" level=info msg="finished
streaming call with code OK"
correlation_id=ccedf938-4c7b-404b-a9da-89e959597626 grpc.code=OK
grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-sidekiq
grpc.method=FindAllBranchNames grpc.request.deadline="2019-05-21T19:12:29Z"
grpc.request.fullMethod=/gitaly.RefService/FindAllBranchNames
grpc.request.glProjectPath=group/project-client
grpc.request.glRepository=project-202
grpc.request.repoPath=group/project-client.git
grpc.request.repoStorage=default grpc.request.topLevelGroup=group
grpc.service=gitaly.RefService grpc.start_time="2019-05-21T19:12:19Z"
grpc.time_ms=3.332 peer.address="10.73.241.140:51252" span.kind=server
system=grpc
2019-05-21_19:12:19.07657 time="2019-05-21T19:12:19Z" level=info msg="finished
unary call with code OK" correlation_id=ccedf938-4c7b-404b-a9da-89e959597626
grpc.code=OK grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-sidekiq
grpc.method=HasLocalBranches grpc.request.deadline="2019-05-21T19:12:29Z"
grpc.request.fullMethod=/gitaly.RepositoryService/HasLocalBranches
grpc.request.glProjectPath=group/project-client
grpc.request.glRepository=project-202
grpc.request.repoPath=group/project-client.git
grpc.request.repoStorage=default grpc.request.topLevelGroup=group
grpc.service=gitaly.RepositoryService grpc.start_time="2019-05-21T19:12:19Z"
grpc.time_ms=3.349 peer.address="10.73.241.140:51252" span.kind=server system=grpc
I'm not able to spot anything specific in the strace file.
Strace files in ticket (for internal use)
Edited by Cindy Pallares 🦉