Corrupted git repositories after restoring from GitLab backup

Summary

After restoring self-hosted GitLab version 14.4.1-ce from a previously prepared backup (on version 14.4.1-ce), all repositories are corrupted:

every repository has a reference refs/heads/main in git file HEAD, even those that haven't main branch;
some repositories have the wrong selected default branch in GitLab GUI (in project settings).

Steps to reproduce

Up new GitLab instance using Docker
Create a new non-empty project
Create a new branch (for example master)
In project settings change the default branch to the new one master
Remove branch main
Make a backup of the whole GitLab, using official instructions
Up new GitLab instance using Docker and restore it using the backup from point 6
Try to clone the project from point 2

What is the current bug behavior?

Impossible to clone project:

$ git clone git@gitlab.home:root/testing_restore.git
Cloning into 'testing_restore'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 3 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
warning: remote HEAD refers to nonexistent ref, unable to checkout.

Because HEAD file contains the wrong reference:

# cat ./git-data/repositories/@hashed/<repo_hash>.git/HEAD
ref: refs/heads/main

What is the expected correct behavior?

The repository state must be equal to the initial state of the repository.

# cat ./git-data/repositories/@hashed/<repo_hash>.git/HEAD
ref: refs/heads/master

Output of checks

Results of GitLab environment info

Used docker-compose.yml

version: "3"

services:
  nginx-proxy:
    image: nginxproxy/nginx-proxy
    container_name: nginx-proxy
    restart: always
    networks:
      - proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/tmp/docker.sock:ro"

  gitlab:
    image: gitlab/gitlab-ce:14.4.1-ce.0
    container_name: ogs-gitlab
    restart: always
    networks:
      - proxy
    environment:
      VIRTUAL_HOST: 192.168.1.10
      VIRTUAL_PORT: 80
      GITLAB_OMNIBUS_CONFIG: |
          external_url 'http://192.168.1.10'
          gitlab_rails['gitlab_shell_ssh_port'] = 22
          nginx['listen_port'] = 80
          nginx['listen_https'] = false
          gitlab_rails['initial_root_password'] = "ykjhfkjhf9t_jhfdhfdFF"
    ports:
        - '22:22'
    volumes:
        - './data/gitlab/config:/etc/gitlab'
        - './data/gitlab/logs:/var/log/gitlab'
        - './data/gitlab/data:/var/opt/gitlab'

networks:
  proxy:

Expand for output related to GitLab environment info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.7.4p191
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.6
Redis Version:  6.0.16
Git Version:    2.33.0.
Sidekiq Version:6.2.2
Go Version:     unknown

GitLab information
Version:        14.4.1
Revision:       1a23d731c9f
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            http://192.168.1.10
HTTP Clone URL: http://192.168.1.10/some-group/some-project.git
SSH Clone URL:  git@192.168.1.10:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        13.21.1
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.21.1 ? ... OK (13.21.1) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 1/2 ... yes Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.4) Git version >= 2.33.0 ? ... yes (2.33.0) Git user has default SSH configuration? ... yes Active users: ... 1 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Quick solution how to make the repositories work

We prepared Python script, that restores HEAD in Git and also in GitLab database using GitLab API. Use it carefully, it's not tested too much.

Edited Nov 23, 2021 by Vladislav I.