Concurrent repository backup inconsistently failing
Summary
During gitaly demo parallel backup froze, only ran on one of two gitaly storages Stacktrace when cancelling job:
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:35:in `join'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:35:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:35:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:105:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:11:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Second run ran other storage successfully, then crashed with:
Circular dependency detected while autoloading constant Storage::Hashed
/opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:2335:in `storage'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/has_repository.rb:16:in `disk_path'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:222:in `display_repo_path'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:186:in `dump_project'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repository.rb:162:in `block (2 levels) in dump_storage'
Steps to reproduce
Run gitlab backup as per concurrent docs https://docs.gitlab.com/ee/raketasks/backup_restore.html#back-up-git-repositories-concurrently
What is the current bug behavior?
Freezes or raises an exception as above.
What is the expected correct behavior?
Completes backup as normal.
Possible fixes
This is likely the result of improperly wrapping rails application code as per https://guides.rubyonrails.org/threading_and_code_execution.html
Edited by James Fargher