Concurrent repository backup inconsistently failing

Summary

During gitaly demo parallel backup froze, only ran on one of two gitaly storages Stacktrace when cancelling job:

/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:35:in `join'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:35:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:35:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:105:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:11:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Second run ran other storage successfully, then crashed with:

Circular dependency detected while autoloading constant Storage::Hashed
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:2335:in `storage'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/has_repository.rb:16:in `disk_path'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:222:in `display_repo_path'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:186:in `dump_project'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:162:in `block (2 levels) in dump_storage'

Steps to reproduce

Run gitlab backup as per concurrent docs https://docs.gitlab.com/ee/raketasks/backup_restore.html#back-up-git-repositories-concurrently

What is the current bug behavior?

Freezes or raises an exception as above.

What is the expected correct behavior?

Completes backup as normal.

Possible fixes

This is likely the result of improperly wrapping rails application code as per https://guides.rubyonrails.org/threading_and_code_execution.html