Push mirror from gitlab.com to secondary site of self-managed GitLab Geo fails

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

When a push mirror is configured in gitlab.com which targets a repo on a self-managed GitLab Geo deployment, if the destination host resolves to the secondary site the mirror sync fails with the following error:

13:get remote references: create git ls-remote: exit status 128, stderr: "fatal: unable to access 'https://gitlab.example.com/group/dest.git/': The requested URL returned error: 302\n".

This is quite likely related to the mirror sync process not handling the redirect that occurs when pushing via a secondary, e.g.

$ git push
warning: redirecting to https://gitlab.exampl.com/-/push_from_secondary/134/group/dest.git/
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 214 bytes | 214.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
remote: 
remote: This request to a Geo secondary node will be forwarded to the
remote: Geo primary node:
remote: 
remote:   https://gitlab.example.com/group/dest.git
remote: 
remote: 
To https://gitlab.example.com/group/dest.git
 * [new branch]      main -> main

Steps to reproduce

  1. Configure GitLab Geo deployment with primary and secondary site sharing unified URL gitlab.example.com
  2. Configure DNS to resolve gitlab.example.com to the IP address of the primary site
  3. Create a project in gitlab.com and configure a push mirror to a project dest in the Geo deployment
  4. Observe mirror sync completes successfully
  5. Update DNS to resolve gitlab.example.com to the IP address of the secondary site
  6. Update a file in the source project and trigger a mirror sync
  7. Observe mirror sync fails with above error

Example Project

What is the current bug behavior?

Push mirror sync only works when target project URL resolves to primary site.

What is the expected correct behavior?

Push mirror sync should work regardless of which Geo site target project URL resolves to.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of: \\\`sudo gitlab-rake gitlab:env:info\\\`) (For installations from source run and paste the output of: \\\`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\\\`)

Results of GitLab application Check

Expand for output related to the GitLab application check 

(For installations with omnibus-gitlab package run and paste the output of: \`sudo gitlab-rake gitlab:check SANITIZE=true\`) (For installations from source run and paste the output of: \`sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true\`) (we will only investigate if the tests are passing)

Possible fixes

Make Workhorse proxy all requests involved in pushes to the primary site (instead of passing them through to Rails and getting a redirect). I'm not experienced with Go, but I'd guess weight 5. I notice Workhorse already has a method getService which can determine if an info-refs request is for a push.

Edited by 🤖 GitLab Bot 🤖