Pushing large repository with huge amount of history causes 500 error

Summary

When I try to push a large (600mb) Git repository with lots of commit history (500,000+ commits) to a new Gitlab repo, I get a 500 Internal Server Error.

Steps to reproduce

Migrate a large Git repository to a new Gitlab repo which has a huge amount of history.

What is the current bug behavior?

Username for 'https://*****': *****
Enumerating objects: 935566, done.
Counting objects: 100% (935566/935566), done.
Delta compression using up to 8 threads
Compressing objects: 100% (128324/128324), done.
Writing objects: 100% (935566/935566), 280.29 MiB | 134.43 MiB/s, done.
Total 935566 (delta 729185), reused 934343 (delta 728158)
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 Internal Server Error
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly
Everything up-to-date

What is the expected correct behavior?

Git push is successful.

Relevant logs and/or screenshots

Gitaly logs

2019-06-11_14:18:29.35331 time="2019-06-11T14:18:29Z" level=info msg="finished streaming call with code Canceled" correlation_id=wFFiMqvgth6 error="rpc error: code = Canceled desc = rpc error: code = Unavailable desc = PostReceivePack: signal: terminated" grpc.code=Canceled grpc.meta.auth_version=v2 grpc.method=PostReceivePack grpc.request.fullMethod=/gitaly.SmartHTTPService/PostReceivePack grpc.request.glProjectPath=*****/duncan grpc.request.glRepository=project-2353 grpc.request.repoPath=612032222/duncan.gi  grpc.request.repoStorage=default grpc.request.topLevelGroup=**** grpc.service=gitaly.SmartHTTPService grpc.start_time="2019-06-11T14:15:49Z" grpc.time_ms=159425.22 peer.address=@ span.kind=server system=grpc

Load Balancer

****** - - [11/Jun/2019:15:38:27 +0100] "GET /gitlab/******/duncan.git/info/refs?service=git-receive-pack HTTP/1.1" 401 421 "-" "git/2.19.1.windows.1"
****** - ******[11/Jun/2019:15:38:35 +0100] "GET /gitlab/******/duncan.git/info/refs?service=git-receive-pack HTTP/1.1" 200 415 "-" "git/2.19.1.windows.1"
****** - ******[11/Jun/2019:15:40:39 +0100] "POST /gitlab/******/duncan.git/git-receive-pack HTTP/1.1" 500 630 "-" "git/2.19.1.windows.1"

production.log

Parameters: {"service"=>"git-receive-pack", "namespace_id"=>"612032222", "project_id"=>"duncan.git"}
{"method":"GET","path":"/gitlab/******/duncan.git/info/refs","format":"*/*","controller":"Projects::GitHttpController","action":"info_refs","status":401,"duration":10.6,"view":0.44,"db":2.27,"time":"2019-06-11T14:38:27.856Z","params":[{"key":"service","value":"git-receive-pack"},{"key":"namespace_id","value":"******"},{"key":"project_id","value":"duncan.git"}],"remote_ip":"******","user_id":null,"username":null,"ua":"git/2.19.1.windows.1","queue_duration":null,"correlation_id":"3ce2b177-6baf-4182-8193-21bac4167735"}
2019-06-11_14:38:27.86810 ******:*** ****** - - [2019/06/11:14:38:27 +0000] "GET /gitlab/******/duncan.git/info/refs?service=git-receive-pack HTTP/1.1" 401 26 "" "git/2.19.1.windows.1" 0.019
172.18.0.1 - - [11/Jun/2019:14:38:27 +0000] "GET /gitlab/******/duncan.git/info/refs?service=git-receive-pack HTTP/1.1" 401 26 "" "git/2.19.1.windows.1"
Started GET "/gitlab/******/duncan.git/info/refs?service=git-receive-pack" for ******at 2019-06-11 14:38:35 +0000
Parameters: {"service"=>"git-receive-pack", "namespace_id"=>"******", "project_id"=>"duncan.git"}
{"method":"GET","path":"/gitlab/******/duncan.git/info/refs","format":"*/*","controller":"Projects::GitHttpController","action":"info_refs","status":200,"duration":145.35,"view":0.32,"db":6.43,"time":"2019-06-11T14:38:35.864Z","params":[{"key":"service","value":"git-receive-pack"},{"key":"namespace_id","value":"******"},{"key":"project_id","value":"duncan.git"}],"remote_ip":"******","user_id":6,"username":"******","ua":"git/2.19.1.windows.1","queue_duration":null,"correlation_id":"cdc0708b-bf23-49c4-ac1a-0f78f4285c43"}
2019-06-11_14:38:36.01789 ******:*** ****** - - [2019/06/11:14:38:36 +0000] "GET /gitlab/******/duncan.git/info/refs?service=git-receive-pack HTTP/1.1" 200 193 "" "git/2.19.1.windows.1" 0.161
******- 612032222 [11/Jun/2019:14:38:36 +0000] "GET /gitlab/******/duncan.git/info/refs?service=git-receive-pack HTTP/1.1" 200 193 "" "git/2.19.1.windows.1"
2019-06-11_14:38:36.01769 time="2019-06-11T14:38:36Z" level=info msg="finished streaming call with code OK" correlation_id=Gk8LTQHJcf1 grpc.code=OK grpc.meta.auth_version=v2 grpc.method=InfoRefsReceivePack grpc.request.fullMethod=/gitaly.SmartHTTPService/InfoRefsReceivePack grpc.request.glProjectPath=******/duncan grpc.request.glRepository=project-2353 grpc.request.repoPath=******/duncan.git grpc.request.repoStorage=default grpc.request.topLevelGroup=******grpc.service=gitaly.SmartHTTPService grpc.start_time="2019-06-11T14:38:36Z" grpc.time_ms=5.789 peer.address=@ span.kind=server system=grpc


Started POST "/gitlab/******/duncan.git/git-receive-pack" for ****** at 2019-06-11 14:39:11 +0000
Parameters: {"namespace_id"=>"******", "project_id"=>"duncan.git"}
{"method":"POST","path":"/gitlab/******/duncan.git/git-receive-pack","format":"application/x-git-receive-pack-result","controller":"Projects::GitHttpController","action":"git_receive_pack","status":200,"duration":216.4,"view":0.34,"db":6.86,"time":"2019-06-11T14:39:11.719Z","params":[{"key":"namespace_id","value":"******"},{"key":"project_id","value":"duncan.git"}],"remote_ip":"******","user_id":6,"username":"******","ua":"git/2.19.1.windows.1","queue_duration":null,"correlation_id":"1f65d178-4b5b-4c8d-8cb8-03d96acbef89"}

The push seems to work on one of our dev environments, but when it comes to pushing to an instance which is used by a large number of users, it fails with the 500 error.