remote rejected on large push (possibly workhorse url issue)
Summary
I have set up gitlab-omnibus v11.4.3-ee, on Debian 9, behind an Apache2 reverse proxy. Everything works perfectly apart when I tried a large (https) push to a new repo and it failed with the following error (some URLs altered for privacy):
Enumerating objects: 3462, done.
Counting objects: 100% (3462/3462), done.
Compressing objects: 100% (3230/3230), done.
Writing objects: 100% (3462/3462), 559.15 MiB | 13.97 MiB/s, done.
Total 3462 (delta 470), reused 0 (delta 0)
remote: Resolving deltas: 100% (470/470), done.
remote: GitLab: API is not accessible
To https://gitlab.server.net/group1/grav_website.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://gitlab.server.net/group1/grav_website.git'
I have found two errors in the logs (which I believe correspond to a timeout at 30s):
==> /var/log/gitlab/gitaly/current <==
2018-10-30_06:20:28.81351 time="2018-10-30T06:20:28Z" level=info msg="finished streaming call" error="rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled" grpc.code=Canceled grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.method=GetNewLFSPointers grpc.request.fullMethod=/gitaly.BlobService/GetNewLFSPointers grpc.request.glRepository=project-6 grpc.request.repoPath=group1/grav_website.git grpc.request.repoStorage=default grpc.request.topLevelGroup=group1 grpc.service=gitaly.BlobService grpc.time_ms=30000 peer.address=@ span.kind=server system=grpc
2018-10-30_06:20:28.84098 time="2018-10-30T06:20:28Z" level=info msg="finished streaming call" grpc.code=OK grpc.meta.auth_version=v2 grpc.method=PostReceivePack grpc.request.fullMethod=/gitaly.SmartHTTPService/PostReceivePack grpc.request.glRepository=project-6 grpc.request.repoPath=group1/grav_website.git grpc.request.repoStorage=default grpc.request.topLevelGroup=group1 grpc.service=gitaly.SmartHTTPService grpc.time_ms=63601 peer.address=@ span.kind=server system=grpc
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
time="2018-10-30T17:20:28+11:00" level=info msg="finished HTTP request" duration=30.082907926 method=POST pid=27198 url="http://127.0.0.1:8080/api/v4/internal/allowed"
time="2018-10-30T17:20:28+11:00" level=error msg="Call failed" body="{\"message\":\"500 Internal Server Error\"}" code=500 method=POST pid=27198 url="http://127.0.0.1:8080/api/v4/internal/allowed"
The call to http://127.0.0.1:8080/api/v4/internal/allowed looks wrong to me as the server is configured to run behind a reverse proxy:
config in: /etc/gitlab/gitlab.rb
external_url 'https://gitlab.server.net'
nginx['listen_port'] = 80
nginx['listen_https'] = false
gitlab_rails['trusted_proxies'] = ['192.168.1.55']
gitlab_rails['mattermost_host'] = "https://mattermost.greenant.net"
mattermost_external_url 'https://mattermost.server.net'
mattermost['service_allowed_untrusted_internal_connections'] = "cach1"
mattermost['service_use_ssl'] = true
mattermost_nginx['listen_port'] = 80
mattermost_nginx['listen_https'] = false
I have checked there is enough RAM (8GB) and storage on the server and all the other processes look OK. I think what might be happening is that the workhorse is triggered and then is inacessible due to the wrong URL. I tried manually setting the workhorse URL to http://gitlab.server.net but then got a redirect loop. My understanding is that it should be getting this setting from external_url anyway.
Steps to reproduce
- attempt push of large number of files.
What is the current bug behavior?
- push is rejected