git push hangs after "remote: Resolving deltas: 100%" with PostReceivePack error on gitlab server
Summary
gitlab version: 13.4.3-ee (fd96f779e9d)
issue description: the git push hangs forever on the last line:
λ git push origin master
Enumerating objects: 267, done.
Counting objects: 100% (229/229), done.
Delta compression using up to 8 threads
Compressing objects: 100% (63/63), done.
Writing objects: 100% (174/174), 49.63 KiB | 5.51 MiB/s, done.
Total 174 (delta 35), reused 162 (delta 24), pack-reused 0
remote: Resolving deltas: 100% (35/35), completed with 12 local objects.Steps to reproduce
the background is that we outsourced the project, so the git repo is on outsource company's gitlab server(TheirRepo.com), and we've recently set up our gitlab server, so that we want to push everything to our repo(OurRepo.com), so first I created an empty private repo through gitlab web portal, then on my local git work folder which now the origin pointing to TheirRepo.com, I did:
git remote rename origin upstream
git remote add origin OurRepo.comnow the origin pointing to our repo, and the upstream pointing to the outsource company. and then I successfully pushed some of the branches onto our gitlab server
git push origin SomeBrancheverything working fine, but when I tried to do the same to master branch, it hangs after printing "remote: Resolving deltas: 100% (35/35), completed with 12 local objects."
PLUS: after did some experiments, I found the direct cause or the actual reproduce method:
if I push master branch first, everything just working fine!
but if I push SomeOtherBranches first, then I'll fail to push master branch!!!
What is the expected correct behavior?
master should also be successfully pushed onto our repo like the other branches
Relevant logs and/or screenshots
after digging around, I found some interesting error log on server, first I found /var/log/gitlab/gitlab-workhorse/current has:
{"correlation_id":"jhMWdD0yFr8","duration_ms":71,"host":"163.16.101.160","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":401,"system":"http","time":"2020-11-25T16:32:22+08:00","uri":"/next-gen/test.git/info/refs?service=git-receive-pack","user_agent":"git/2.29.2.windows.2","written_bytes":26}
{"correlation_id":"VipD5p5KeD5","duration_ms":280,"host":"163.16.101.160","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2020-11-25T16:32:23+08:00","uri":"/next-gen/test.git/info/refs?service=git-receive-pack","user_agent":"git/2.29.2.windows.2","written_bytes":246}
{"correlation_id":"hwINl0fcUg8","duration_ms":105,"host":"163.16.101.160","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2020-11-25T16:32:25+08:00","uri":"/api/v4/internal/allowed","user_agent":"Go-http-client/1.1","written_bytes":681}
{"correlation_id":"4rLOcmiM9Y4","duration_ms":150,"host":"163.16.101.160","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2020-11-25T16:32:25+08:00","uri":"/api/v4/internal/pre_receive","user_agent":"Go-http-client/1.1","written_bytes":36}
{"correlation_id":"CRxXsx22P13","error":"handleReceivePack: smarthttp.ReceivePack: rpc error: code = Internal desc = failed proxying to secondary: rpc error: code = Unavailable desc = PostReceivePack: exit status 128","level":"error","method":"POST","msg":"error","time":"2020-11-25T16:32:25+08:00","uri":"/next-gen/test.git/git-receive-pack"}
{"correlation_id":"CRxXsx22P13","duration_ms":548,"host":"163.16.101.160","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2020-11-25T16:32:25+08:00","uri":"/next-gen/test.git/git-receive-pack","user_agent":"git/2.29.2.windows.2","written_bytes":703}
and then I found error on gitaly cluster server: /var/log/gitlab/gitaly/current
{"error":"error voting on transaction: error voting on transaction: transaction was aborted","grpc.code":"Internal","grpc.meta.auth_version":"v2","grpc.meta.deadline_type":"none","grpc.method":"ReferenceTransactionHook","grpc.request.fullMethod":"/gitaly.HookService/ReferenceTransactionHook","grpc.request.glProjectPath":"/next-gen/test","grpc.request.glRepository":"project-11","grpc.request.repoPath":"@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8.git","grpc.request.repoStorage":"gitaly-1","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.HookService","grpc.start_time":"2020-11-26T01:22:48Z","grpc.time_ms":14.148,"level":"error","msg":"finished streaming call with code Internal","peer.address":"@","pid":6702,"span.kind":"server","system":"grpc","time":"2020-11-26T01:22:48.905Z"}
{"error":"error voting on transaction: error voting on transaction: rpc error: code = Internal desc = subtransaction has failed","grpc.code":"Internal","grpc.meta.auth_version":"v2","grpc.meta.deadline_type":"none","grpc.method":"ReferenceTransactionHook","grpc.request.fullMethod":"/gitaly.HookService/ReferenceTransactionHook","grpc.request.glProjectPath":"/next-gen/test","grpc.request.glRepository":"project-11","grpc.request.repoPath":"@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8.git","grpc.request.repoStorage":"gitaly-1","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.HookService","grpc.start_time":"2020-11-26T01:22:48Z","grpc.time_ms":1.25,"level":"error","msg":"finished streaming call with code Internal","peer.address":"@","pid":6702,"span.kind":"server","system":"grpc","time":"2020-11-26T01:22:48.919Z"}
{"correlation_id":"CqyeiP2Iyc8","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-workhorse","grpc.meta.deadline_type":"none","grpc.method":"PostReceivePack","grpc.request.fullMethod":"/gitaly.SmartHTTPService/PostReceivePack","grpc.request.glProjectPath":"/next-gen/test","grpc.request.glRepository":"project-11","grpc.request.repoPath":"@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8.git","grpc.request.repoStorage":"gitaly-1","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.SmartHTTPService","grpc.start_time":"2020-11-26T01:22:48Z","level":"error","msg":"error executing git hookerror executing git hookfatal: ref updates aborted by hook\\n","peer.address":"163.16.101.162:39070","pid":6702,"span.kind":"server","system":"grpc","time":"2020-11-26T01:22:48.920Z"}
{"correlation_id":"CqyeiP2Iyc8","diskcache":"3764ffdd-944a-4977-b2ee-faa1360827ca","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-workhorse","grpc.meta.deadline_type":"none","grpc.method":"PostReceivePack","grpc.request.fullMethod":"/gitaly.SmartHTTPService/PostReceivePack","grpc.request.glProjectPath":"/next-gen/test","grpc.request.glRepository":"project-11","grpc.request.repoPath":"@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8.git","grpc.request.repoStorage":"gitaly-1","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.SmartHTTPService","grpc.start_time":"2020-11-26T01:22:48Z","level":"info","msg":"diskcache state change","peer.address":"163.16.101.162:39070","pid":6702,"span.kind":"server","system":"grpc","time":"2020-11-26T01:22:48.922Z"}
I searched using the keyword from the error log, but no luck, please help provide some opinions or troubleshooting methods, thanks
Edited by Liu Yue