GitLab backup restore - rake aborted! - ActiveRecord::AssociationTypeMismatch: Project expected, got ForkNetwork

Summary

GitLab restore fails during restore object pools phase due to AssociationTypeMismatch error when PoolRepository.source_project is nil.

Steps to reproduce

I didn't verify the steps on separate environment, but this is what I think are required steps in high level:

  1. Have PoolRepository object which source_project is nil in the database, for example: #<PoolRepository id:1 state:ready disk_path:@pools/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b source_project: nil>
  2. Also PoolRepository.member_projects.first&.root_of_fork_network should return related ForkNetWork and not nil
  3. Take GitLab backup, for example sudo gitlab-backup create BACKUP=dump GZIP_RSYNCABLE=yes STRATEGY=copy
  4. Restore the backup: sudo GITLAB_ASSUME_YES=1 gitlab-backup restore
  5. Restore fails due to the error

Regarding step 1 and 2, I have no clue how our self-managed GitLab instance achieved that state.

Example Project

What is the current bug behavior?

gitlab-backup restore fails to complete during restore object pools phase and it exits with status 1. The restore cannot be completed without modifying backup or GitLab source codes.

What is the expected correct behavior?

gitlab-backup restore should succesfully restore all data and report Restore task is done..

Relevant logs and/or screenshots

Stacktrace/error output from gitlab-backup restore command:

rake aborted!
ActiveRecord::AssociationTypeMismatch: Project(#208440) expected, got #<ForkNetwork id: 78, root_project_id: 707, deleted_root_project_name: nil> which is an instance of ForkNetwork(#219540)
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repositories.rb:189:in `block in restore_object_pools'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repositories.rb:186:in `restore_object_pools'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repositories.rb:66:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:160:in `run_restore_task'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:127:in `block in restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:126:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:126:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:20:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:restore
(See full trace by running task with --trace)
 - Object pool @pools/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b...
Transfering ownership of /var/opt/gitlab/gitlab-rails/shared/registry to registry
Command exited with non-zero status 1

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

Replaced real hostname with my-hostname.

System information
System:		Debian 11
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.7.5p203
Gem Version:	3.1.4
Bundler Version:2.2.33
Rake Version:	13.0.6
Redis Version:	6.2.6
Sidekiq Version:6.4.0
Go Version:	unknown

GitLab information
Version:	14.9.4-ee
Revision:	58f08854bfb
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.7
URL:		https://my-hostname
HTTP Clone URL:	https://my-hostname/some-group/some-project.git
SSH Clone URL:	git@my-hostname:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	yes
Using Omniauth:	no

GitLab Shell
Version:	13.24.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check

I executed the gitlab:check command after fixing the bug and running restore succesfully, but here it is.

On Projects have namespace: ... step I removed rest projects after 3/2 ... yes. All of them had yes.

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.24.0 ? ... OK (13.24.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
	User output sanitized. Found 100 users of 100 limit.

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ... 
3/2 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git user has default SSH configuration? ... yes
Active users: ... 443
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Buggy line of code in version 14.9.4. It looks like this:

pool.source_project ||= pool.member_projects.first&.root_of_fork_network

And can be fixed:

pool.source_project ||= Project.find_by_id(pool.member_projects.first&.root_of_fork_network&.root_project_id)