repositories restore fails using backup-utility

Summary

Unable to restore repositories using backup-utility --restore: restore fails for each repo

Steps to reproduce

• install Gitlab Helm Chart
• create a Gitlab group /mygroup
• create a "blanck project with README" /mygroup/myproject
• execute backup-utility -t sample on task-runner pod
• execute backup-utility --restore -t sample

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

# fragment
# praefect activated

global:
 praefect:
    enabled: true
    virtualStorages:
    - name: default
      gitalyReplicas: 4
      maxUnavailable: 1

gitlab:
  gitlab:
    praefect:
      metrics:
        metricsPort: "9236"
      logging:
        format: "json"
        level: "info"
      electionStrategy: "per_repository"

Current behavior

Restore fails with error on each repo:

`error="manager: remove repository: rpc error: code = NotFound desc = mutator call: route repository mutator: get primary:`

Expected behavior

Gitlab fully restored

Versions

• Chart: v5.3.2
• Platform:
  • Self-hosted: Pivotal Kubernetes Service
• Kubernetes: (kubectl version)
  • Client: 1.21.2
  • Server: 1.19.6
• Helm: (helm version)
  • Client: v3.6.1

Relevant logs

backup-utility --restore

021-10-12 14:25:41 +0000 -- Restoring repositories ...
time="2021-10-12T14:25:42Z" level=info msg="started restore" command=restore gl_project_path=gitlab-instance-5873fbe9.wiki relative_path=@groups/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git storage_name=default
time="2021-10-12T14:25:42Z" level=error msg="restore failed" command=restore error="manager: remove repository: rpc error: code = NotFound desc = mutator call: route repository mutator: get primary: repository \"default\"/\"@groups/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git\" not found" gl_project_path=gitlab-instance-5873fbe9.wiki relative_path=@groups/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git storage_name=default
time="2021-10-12T14:25:42Z" level=info msg="started restore" command=restore gl_project_path=mygroup.wiki relative_path=@groups/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.wiki.git storage_name=default
time="2021-10-12T14:25:42Z" level=error msg="restore failed" command=restore error="manager: remove repository: rpc error: code = NotFound desc = mutator call: route repository mutator: get primary: repository \"default\"/\"@groups/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.wiki.git\" not found" gl_project_path=mygroup.wiki relative_path=@groups/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.wiki.git storage_name=default
time="2021-10-12T14:25:43Z" level=info msg="started restore" command=restore gl_project_path=gitlab-instance-5873fbe9/Monitoring relative_path=@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git storage_name=default
time="2021-10-12T14:25:43Z" level=info msg="started restore" command=restore gl_project_path=gitlab-instance-5873fbe9/Monitoring.wiki relative_path=@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git storage_name=default
time="2021-10-12T14:25:43Z" level=error msg="restore failed" command=restore error="manager: remove repository: rpc error: code = NotFound desc = mutator call: route repository mutator: get primary: repository \"default\"/\"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git\" not found" gl_project_path=gitlab-instance-5873fbe9/Monitoring.wiki relative_path=@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git storage_name=default
time="2021-10-12T14:25:43Z" level=info msg="started restore" command=restore gl_project_path=gitlab-instance-5873fbe9/Monitoring relative_path=@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git storage_name=default
time="2021-10-12T14:25:43Z" level=error msg="restore failed" command=restore error="manager: remove repository: rpc error: code = NotFound desc = mutator call: route repository mutator: get primary: repository \"default\"/\"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git\" not found" gl_project_path=gitlab-instance-5873fbe9/Monitoring relative_path=@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git storage_name=default
time="2021-10-12T14:25:43Z" level=info msg="started restore" command=restore gl_project_path=mygroup/myproject relative_path=@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git storage_name=default
time="2021-10-12T14:25:43Z" level=info msg="completed restore" command=restore gl_project_path=gitlab-instance-5873fbe9/Monitoring relative_path=@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git storage_name=default
time="2021-10-12T14:25:43Z" level=info msg="started restore" command=restore gl_project_path=mygroup/myproject.wiki relative_path=@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git storage_name=default
time="2021-10-12T14:25:43Z" level=error msg="restore failed" command=restore error="manager: remove repository: rpc error: code = NotFound desc = mutator call: route repository mutator: get primary: repository \"default\"/\"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git\" not found" gl_project_path=mygroup/myproject.wiki relative_path=@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.wiki.git storage_name=default
time="2021-10-12T14:25:43Z" level=info msg="started restore" command=restore gl_project_path=mygroup/myproject relative_path=@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.design.git storage_name=default
time="2021-10-12T14:25:43Z" level=error msg="restore failed" command=restore error="manager: remove repository: rpc error: code = NotFound desc = mutator call: route repository mutator: get primary: repository \"default\"/\"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.design.git\" not found" gl_project_path=mygroup/myproject relative_path=@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.design.git storage_name=default
time="2021-10-12T14:25:44Z" level=info msg="completed restore" command=restore gl_project_path=mygroup/myproject relative_path=@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git storage_name=default
time="2021-10-12T14:25:44Z" level=fatal msg="restore: pipeline: 6 failures encountered"
rake aborted!
Backup::Error: gitaly-backup exit status 1
/srv/gitlab/lib/backup/gitaly_backup.rb:44:in `wait'
/srv/gitlab/lib/backup/repositories.rb:51:in `restore'
/srv/gitlab/lib/tasks/gitlab/backup.rake:124:in `block (4 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:5:in `load'
/srv/gitlab/bin/bundle:5:in `<main>'
Tasks: TOP => gitlab:backup:repo:restore
(See full trace by running task with --trace)

(Please provide any relevant log snippets you have collected, using code blocks (```) to format)