Missing branches in GitLab 9.3 after a push

Customer is running an HA setup for their primaries and GitLab Geo secondaries in two different continents.

They noticed after a push, master and other local branches were wiped from the local branch.

They are using GitLab 9.3, which has system hooks firing even for the primary node due to a bug that was fixed in 9.4. Because of their HA setup, we see that after a push, this causes a system hook to fire for the push update, which is then received by one of the HA nodes. As a result, we see these messages on nodes in the primary cluster:

2017-10-27T15:54:16-07:00 gitlab-node gitlab-shell.log: I, [2017-10-27T15:54:16.647241 #13277]  INFO -- : Fetching remote geo for project group/project.git

Looking at the repository in question on disk, we do see a remote for github for a previous GitHub import, as well as geo pointing to the Geo master node.

Possible ideas what might be happening, but no hard evidence yet for any of these ideas:

1. The fetch for the Geo runs git fetch geo --prune, which possibly wipes out local branches. However, as far as I can tell the --prune option only deletes remote tracking branches, not local branches.
2. RemoteUpdateMirrorWorker is running, which tries to sync against the old GitHub repository. We saw no evidence this is happening in Sidekiq. It also seems unlikely that master would not be present in GitHub, but we don't have access to confirm.
3. One of the nodes in the cluster has a bad repository? A quick check showed no evidence of this.
4. Race condition with creating a packed-refs with git gc file and a new branch?

Note this system hook bug was fixed in 9.4 in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1679, and system hooks were removed completely in 10.0.

ZD: https://gitlab.zendesk.com/agent/tickets/84168

/cc: @brodock, @nick.thomas, @lbot