Repository#fetch_geo_mirror quietly fails if the repository is broken

ZD: https://gitlab.zendesk.com/agent/tickets/71280

Somehow one of the customer's repositories got into a bad state. We ran GIT_TRACE=1 git fetch geo manually, and it didn't appear to do anything.

Then @MrChrisW had the good idea of trying to clone the bare repo, and this is what we got:

image

Because this repo was in a broken state, Geo was unable to update.

How did the repo wind up in this state? Possible reasons: bad initial rsync, aborted bad git fetch?

We need to:

  1. Check the return value of Repository#fetch_geo_mirror.
  2. Log an error to a specific place. This needs to be saved in a place where it's easy to look. Ideally also we can send this to the primary.
  3. Flag this repo as unhealthy.
  4. Design some recovery mechanism; consider rewiping and recloning if necesssary