Deleting remote branches via Geo secondary git proxy hangs when using SSH

This issue was separated out of #322519 (closed) as those are two completely independent issues.

This issue is valid for all the versions since Geo secondary git push proxy feature introduction.

If a user wants to delete a remote branch via a Geo secondary, they would normally type git push <REMOTE> --delete <BRANCH>. When this is executed against a secondary node, via SSH - the command freezes up, and it is necessary for the user to press CTRL+C to regain control. The branch however is deleted afterward.

There is no obvious errors in the server logs.

Steps to reproduce

Setup GitLab Geo with 1 primary node and 1 secondary node.

The example below assumes the following roles:

geo1 (geo1.vagrant) - primary
geo2 (geo2.vagrant) - secondary

1. Create a test project with 2 branches (alpha and beta).

2. Clone the project locally with SSH. Setup remotes geo1 and geo2.

3. Run the following commands. Notice that the alpha branch is deleted when executed against geo1, but when deleting the beta branch against geo2, the command hangs. Pressing CTRL+C gets around the hang, and you can see that the branch has been deleted in GitLab.

   ❯ git remote -v
   geo1	git@geo1.vagrant:root/delete-test.git (fetch)
   geo1	git@geo1.vagrant:root/delete-test.git (push)
   geo2	git@geo2.vagrant:root/delete-test.git (fetch)
   geo2	git@geo2.vagrant:root/delete-test.git (push)
   
   ❯ git fetch --all
   Fetching geo1
   From geo1.vagrant:root/delete-test
   * [new branch]      alpha      -> geo1/alpha
   * [new branch]      beta       -> geo1/beta
   * [new branch]      master     -> geo1/master
   Fetching geo2
   From geo2.vagrant:root/delete-test
   * [new branch]      alpha      -> geo2/alpha
   * [new branch]      beta       -> geo2/beta
   * [new branch]      master     -> geo2/master
   
   ❯ git push geo1 --delete alpha
   To geo1.vagrant:root/delete-test.git
   - [deleted]         alpha
   
   ❯ git push geo2 --delete beta
   remote:
   remote: This request to a Geo secondary node will be forwarded to the
   remote: Geo primary node:
   remote:
   remote:   git@geo1.vagrant:root/delete-test.git
   remote:
   ^C

What is the current bug behavior?

When deleting remote branches via a Geo secondary node, either the command hangs or doesn't complete successfully. As a result, the branch is deleted after pressing CTRL+C (for SSH) or the command fails completely, and the branch is not deleted (HTTP).

What is the expected correct behavior?

When deleting remote branches via a Geo secondary node, the commands should be successful.

Geo SSH proxy push diagram

sequenceDiagram
  participant Client
  participant Secondary
  participant SecondaryGitLabShell as Secondary gitlab-shell
  participant Primary
  Note over Client,Secondary: $ git push
  Client->>SecondaryGitLabShell: ssh <single> git-receive-pack ''/repo.git''
  SecondaryGitLabShell->>+Secondary: HTTP POST /api/v4/internal/allowed
  Secondary-->>-SecondaryGitLabShell: OK, REDIRECT
  SecondaryGitLabShell->>+Secondary: HTTP POST /api/v4/geo/proxy_git_push_ssh/info_refs
  Secondary-->>Primary: HTTP GET /repo.git/info/refs?service=git-receive-pack
  Primary-->>Secondary: <refs>
  Secondary->>-SecondaryGitLabShell: <refs>
  SecondaryGitLabShell->>+Secondary: HTTP POST /api/v4/geo/proxy_git_push_ssh/push
  Secondary-->>Primary: HTTP POST /repo.git/info/refs?service=git-receive-pack/git-receive-pack
  Primary-->>Secondary: <refs>
  Secondary->>-SecondaryGitLabShell: <refs>
  SecondaryGitLabShell-->>+Client: <refs>