Bug: Git push over SSH against a secondary Geo site has 1.86GB limit
Summary
When pushing a new file over 1GB to a secondary Geo site over SSH, the push fails at 1GB:
Note: After #391407 (closed), the push fails at 1.86GB.
Steps to reproduce
More details: #391407 (comment 1406524065)
Example Project
What is the current bug behavior?
When pushing a new file over 1.86GB to a secondary Geo site over SSH, the push consistently fails at 1.86GB.
What is the expected correct behavior?
When pushing a large file, it should not fail, regardless of the size of the push. 1.86GB is clearly too limited.
Relevant logs and/or screenshots
➜ pushoversshtosecondary git:(main) time GIT_TRACE=1 git push
11:39:04.881772 git.c:439 trace: built-in: git push
11:39:04.883021 run-command.c:655 trace: run_command: unset GIT_PREFIX; ssh -p 22 git@geo.staging-ref.gitlab.com 'git-receive-pack '\''/mkozono/pushoversshtosecondary.git'\'''
remote:
remote: This request to a Geo secondary node will be forwarded to the
remote: Geo primary node:
remote:
remote: git@staging-ref.gitlab.com:mkozono/pushoversshtosecondary.git
remote:
11:39:09.670583 run-command.c:655 trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
11:39:09.684266 git.c:439 trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 10 threads
Compressing objects: 100% (2/2), done.
send-pack: unexpected disconnect while reading sideband packet
Connection to geo.staging-ref.gitlab.com closed by remote host.
fatal: the remote end hung up unexpectedly
GIT_TRACE=1 git push 18.95s user 7.69s system 6% cpu 7:04.26 total
20230525-staging-ref-logs-during-1GB-failure.json
I also tested this with the feature flag geo_proxy_direct_to_primary
enabled. It failed at 1.86GB instead of 1GB:
Click here to expand the similar Git push failures
➜ pushoversshtosecondary git:(main) time GIT_TRACE=1 git push
09:56:44.609617 git.c:439 trace: built-in: git push
09:56:44.610576 run-command.c:655 trace: run_command: unset GIT_PREFIX; ssh -p 22 git@geo.staging-ref.gitlab.com 'git-receive-pack '\''/mkozono/pushoversshtosecondary.git'\'''
remote:
remote: This request to a Geo secondary node will be forwarded to the
remote: Geo primary node:
remote:
remote: git@staging-ref.gitlab.com:mkozono/pushoversshtosecondary.git
remote:
09:56:52.096509 run-command.c:655 trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
09:56:52.119641 git.c:439 trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 10 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 1.86 GiB | 702.00 KiB/s, done.
Total 3 (delta 1), reused 1 (delta 0), pack-reused 0
remote:
remote: ========================================================================
remote:
remote: ERROR: Remote repository is unavailable
remote:
remote: ========================================================================
remote:
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
GIT_TRACE=1 git push 43.93s user 26.88s system 2% cpu 46:34.32 total
➜ pushoversshtosecondary git:(main) time GIT_TRACE=1 git push
10:51:34.660529 git.c:439 trace: built-in: git push
10:51:34.661983 run-command.c:655 trace: run_command: unset GIT_PREFIX; ssh -p 22 git@geo.staging-ref.gitlab.com 'git-receive-pack '\''/mkozono/pushoversshtosecondary.git'\'''
remote:
remote: This request to a Geo secondary node will be forwarded to the
remote: Geo primary node:
remote:
remote: git@staging-ref.gitlab.com:mkozono/pushoversshtosecondary.git
remote:
10:51:39.693275 run-command.c:655 trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
10:51:39.713926 git.c:439 trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 10 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 1.86 GiB | 1.70 MiB/s, done.
Total 3 (delta 1), reused 1 (delta 0), pack-reused 0
remote:
remote: ========================================================================
remote:
remote: ERROR: Remote repository is unavailable
remote:
remote: ========================================================================
remote:
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
GIT_TRACE=1 git push 17.77s user 5.38s system 2% cpu 18:49.60 total
Here are the logs on staging-ref during both 1.86GB failures, scoped to the name of the project pushoversshtosecondary
:
20230525-staging-ref-logs-during-1.86GB-failure.json
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)