Support restoring repository backups in parallel

Merged James Fargher requested to merge parallel_backup_restore into master

What does this MR do?

#35862 (closed)

Concurrent restores were added to gitaly-backup in gitaly!3796 (merged) without also changing gitlab-rails this concurrency would not be controllable.

How to setup and validate locally (strongly suggested)

  1. Make sure GDK is up-to-date. Without the requisite changes in gitaly, trying to restore with concurrency will result in an error.
  2. Take a backup (shell):
    $ bundle exec rake gitlab:backup:create
    ...
    Creating backup archive: 1630989194_2021_09_07_14.3.0-pre_gitlab_backup.tar ... done
  3. Restore the backup with concurrency (shell):
    $ be rake gitlab:backup:restore GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1 BACKUP=1630989194_2021_09_07_14.3.0-pre
    If repository restores begin, the change has succeeded.

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by James Fargher