Deleting remote branches via Geo secondary git proxy hangs (SSH) or fails (HTTP)

Summary

Tested on 13.9.0-ee

If a user wants to delete a remote branch via a Geo secondary, they would normally type git push <REMOTE> --delete <BRANCH>.

When this is executed against a secondary node:

• Via SSH - the command freezes up, and it is necessary for the user to press CTRL+C to regain control. The branch however is deleted afterwards.
• Via HTTP - the command fails immediately, but the branch is not deleted.

There is no obvious errors in the server logs.

Steps to reproduce

Setup GitLab Geo with 1 primary node and 1 secondary node.

The example below assumes the following roles:

geo1 (geo1.vagrant) - primary
geo2 (geo2.vagrant) - secondary

For SSH:

1. Create a test project with 2 branches (alpha and beta).

2. Clone the project locally with SSH. Setup remotes geo1 and geo2.

3. Run the following commands. Notice that the alpha branch is deleted when executed against geo1, but when deleting the beta branch against geo2, the command hangs. Pressing CTRL+C gets around the hang, and you can see that the branch has been deleted in GitLab.

   ❯ git remote -v
   geo1	git@geo1.vagrant:root/delete-test.git (fetch)
   geo1	git@geo1.vagrant:root/delete-test.git (push)
   geo2	git@geo2.vagrant:root/delete-test.git (fetch)
   geo2	git@geo2.vagrant:root/delete-test.git (push)
   
   ❯ git fetch --all
   Fetching geo1
   From geo1.vagrant:root/delete-test
   * [new branch]      alpha      -> geo1/alpha
   * [new branch]      beta       -> geo1/beta
   * [new branch]      master     -> geo1/master
   Fetching geo2
   From geo2.vagrant:root/delete-test
   * [new branch]      alpha      -> geo2/alpha
   * [new branch]      beta       -> geo2/beta
   * [new branch]      master     -> geo2/master
   
   ❯ git push geo1 --delete alpha
   To geo1.vagrant:root/delete-test.git
   - [deleted]         alpha
   
   ❯ git push geo2 --delete beta
   remote:
   remote: This request to a Geo secondary node will be forwarded to the
   remote: Geo primary node:
   remote:
   remote:   git@geo1.vagrant:root/delete-test.git
   remote:
   ^C

For HTTP:

1. Create a test project with 2 branches (gamma and delta).

2. Clone the project locally with SSH. Setup remotes geo1 and geo2.

3. Run the following commands. Notice that the gamma branch is deleted when executed against geo1, but when deleting the delta branch against geo2, the command errors out immediately. The branch is not deleted.

   ❯ git remote -v
   geo1	http://root:<SANITIZED>@geo1.vagrant/root/delete-test.git (fetch)
   geo1	http://root:<SANITIZED>@geo1.vagrant/root/delete-test.git (push)
   geo2	http://root:<SANITIZED>@geo2.vagrant/root/delete-test.git (fetch)
   geo2	http://root:<SANITIZED>@geo2.vagrant/root/delete-test.git (push)
   
   ❯ git fetch --all
   Fetching geo1
   From http://geo1.vagrant/root/delete-test
   * [new branch]      delta      -> geo1/delta
   * [new branch]      gamma      -> geo1/gamma
   * [new branch]      master     -> geo1/master
   Fetching geo2
   From http://geo2.vagrant/root/delete-test
   * [new branch]      delta      -> geo2/delta
   * [new branch]      gamma      -> geo2/gamma
   * [new branch]      master     -> geo2/master
   
   ❯ git push geo1 --delete gamma
   To http://geo1.vagrant/root/delete-test.git
   - [deleted]         gamma
   
   ❯ git push geo2 --delete delta
   remote: HTTP Basic: Access denied
   fatal: Authentication failed for 'http://geo2.vagrant/root/delete-test.git/'

What is the current bug behavior?

When deleting remote branches via a Geo secondary node, either the command hangs or doesn't complete successfully. As a result, the branch is deleted after pressing CTRL+C (for SSH) or the command fails completely, and the branch is not deleted (HTTP).

What is the expected correct behavior?

When deleting remote branches via a Geo secondary node, the commands should be successful.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes