Missing branches in GitLab 9.3 after a push
Customer is running an HA setup for their primaries and GitLab Geo secondaries in two different continents.
They noticed after a push, master
and other local branches were wiped from the local branch.
They are using GitLab 9.3, which has system hooks firing even for the primary node due to a bug that was fixed in 9.4. Because of their HA setup, we see that after a push, this causes a system hook to fire for the push update, which is then received by one of the HA nodes. As a result, we see these messages on nodes in the primary cluster:
2017-10-27T15:54:16-07:00 gitlab-node gitlab-shell.log: I, [2017-10-27T15:54:16.647241 #13277] INFO -- : Fetching remote geo for project group/project.git
Looking at the repository in question on disk, we do see a remote for github
for a previous GitHub import, as well as geo
pointing to the Geo master node.
Possible ideas what might be happening, but no hard evidence yet for any of these ideas:
- The fetch for the Geo runs
git fetch geo --prune
, which possibly wipes out local branches. However, as far as I can tell the--prune
option only deletes remote tracking branches, not local branches. - RemoteUpdateMirrorWorker is running, which tries to sync against the old GitHub repository. We saw no evidence this is happening in Sidekiq. It also seems unlikely that
master
would not be present in GitHub, but we don't have access to confirm. - One of the nodes in the cluster has a bad repository? A quick check showed no evidence of this.
- Race condition with creating a
packed-refs
withgit gc
file and a new branch?
Note this system hook bug was fixed in 9.4 in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1679, and system hooks were removed completely in 10.0.
ZD: https://gitlab.zendesk.com/agent/tickets/84168
/cc: @brodock, @nick.thomas, @lbot