Incorrect branch assigned as default when restoring new GitLab Omnibus instance from backup

Summary

GitLab assigns an incorrect branch as the default when restoring a new GitLab instance from backup. This seems to affect all projects, whether the default branch was set on the instance or project level. The original instance does not have a master or main branch, and the default branch on the backed-up instance is defined.

The branch designated as default seems to be selected by alphanumeric order. This issue affects new and imported projects.

Steps to reproduce

Below were my steps to reproduce

First instance
  1. Spun up new GitLab 14.4.1 instance
  2. Changed the default branch on the instance level. -- In my case, I chose the name zed so that it falls at the bottom of the alphanumeric list
  3. Created a new project
  4. Added several branches to new project with names beginning with various letters and numbers -- confirmed that no master or main branch is present in new project
  5. Imported an existing project with numerous pre-existing branches
  6. Deleted master branch, confirmed main does not exist
  7. Created a new branch of the name zed and assigned it as the default on the project level of the imported project
  8. Followed documented backup procedure to back up git data, as well as secrets and gitlab.rb
Second instance
  1. Spun up new Omnibus 14.1.1 instance
  2. Followed restore documentation to restore from backup, including secrets and rb
  3. Observed that the default branch on both the new and the imported project had been assigned based on alphanumeric order.
  4. The default branch on the project level shows these new branches instead of zed, despite zed still existing on both projects. The instance-level default in the admin area is still set to zed as it was on the previous instance.

No migrations were in process when the backup was taken.

Example Project

These projects are on a GCP instance that is not always on; devs, please feel free to contact me and we can pair on this and I can show and reproduce for you.

What is the current bug behavior?

The default branch assigned on projects is not honored when restoring from backup on a new instance and is instead assigned by alphanumeric order.

What is the expected correct behavior?

Default branch settings should be honored and imported after a restore from backup.

Relevant logs and/or screenshots

My colleague was able to identify the branch-choosing logic within the code, but I don't code enough to offer a fix:

There's a fair amount of logic around this
- If there's one branch in the repo we return that https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/git/localrepo/refs.go#L280
- Then we check if HEAD matches a branch, then for main and master https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/git/localrepo/refs.go#L291-303
- Then if nothing else matched we just return the first branch name we find https://gitlab.com/gitlab-org/gitaly/-/blob/master/internal/git/localrepo/refs.go#L314

Hopefully this helps isolate the cause of the issue. 😃

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Edited by Brad Sevy