Repositories gone after update. Broken migration to hashed storage.

Summary

Apologies if this has been reported elsewhere. I didn't see anything similar after some searching. I'm reporting the issue being discussed in this forum thread. After upgrading my omnibus installation to 13.4.1 all but 3 of my repositories are inaccessible. I see a message that says:

No repository

The repository for this project does not exist.

Relevant logs and/or screenshots

After reading the forum thread, I understand that the issue has to do with the migration from legacy storage to hashed storage. It looks like 3 of my projects successfully made the transition and the remainder did not.

Here is a screenshot showing some failed migration jobs:

They all fail with the same error:

Gitlab::Git::CommandError: 2:NoMethodError: undefined method `relative_path’ for nil:NilClass.

The same error shows up if I try to manually run the migrations using the gitlab-rake commands shown here.

Interestingly, if I use the rake commands to list which projects are on hashed storage vs legacy storage I see that the 3 repositories that I can access are listed as on hashed storage and all of the rest are still on legacy storage.

Results of GitLab environment info

Expand for output related to GitLab environment info

# gitlab-rake gitlab:env:info
System information
System:		Ubuntu 18.04
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.28.0
Sidekiq Version:5.2.9
Go Version:	unknown
GitLab information
Version:	13.4.1
Revision:	c90be62bdef
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:
GitLab Shell
Version:	13.7.0
Repository storage paths:
default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.7.0 ? ... OK (13.7.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/1 ... yes
3/2 ... yes
4/3 ... yes
4/4 ... yes
5/5 ... yes
5/6 ... yes
5/7 ... yes
5/8 ... yes
6/9 ... yes
5/10 ... yes
4/12 ... yes
5/13 ... yes
5/14 ... yes
5/15 ... yes
2/16 ... yes
5/17 ... yes
5/18 ... yes
5/19 ... yes
5/20 ... yes
5/21 ... yes
5/22 ... yes
5/23 ... yes
5/24 ... yes
5/25 ... yes
5/26 ... yes
5/27 ... yes
5/28 ... yes
5/29 ... yes
2/30 ... yes
5/31 ... yes
5/32 ... yes
5/33 ... yes
5/34 ... yes
2/35 ... yes
5/36 ... yes
5/37 ... yes
5/38 ... yes
5/39 ... yes
5/40 ... yes
5/41 ... yes
5/42 ... yes
5/43 ... yes
5/44 ... yes
2/45 ... yes
5/46 ... yes
5/47 ... yes
5/48 ... yes
5/49 ... yes
3/50 ... yes
5/51 ... yes
5/53 ... yes
9/54 ... yes
5/55 ... yes
5/56 ... yes
5/58 ... yes
5/59 ... yes
5/60 ... yes
5/61 ... yes
2/62 ... yes
5/63 ... yes
5/64 ... yes
5/65 ... yes
5/66 ... yes
5/67 ... yes
5/68 ... yes
5/69 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.24.0 ? ... yes (2.28.0)
Git user has default SSH configuration? ... yes
Active users: ... 3
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... no
Try fixing it:
Please migrate all projects to hashed storage
as legacy storage is deprecated in 13.0 and support will be removed in 14.0.
For more information see:
doc/administration/repository_storage_types.md
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Workaround

This snippet will fix any repositories that are broken as a result of the migration.