Allow backing up or restoring of repositories in a specific storage

Merged James Fargher requested to merge backup_storages into master

What does this MR do and why?


Adds a new option REPOSITORIES_STORAGES to backup.rake. This accepts a comma separated list of storages, only these storages will be used to create or restore repositories.

How to set up and validate locally

  1. Update gdk.yml to add more storages:
      storage_count: 2
  2. Rejigger gdk:
    $ gdk reconfigure && gdk restart
  3. Move some repo to the new storage, it will be called gitaly-1.
    [1] pry(main)> move = Project.find(21) source_storage_name: 'default', destination_storage_name: 'gitaly-1'
    [2] pry(main)> move.schedule
    => true
  4. Take a backup of only this new storage:
    $ bundle exec rake gitlab:backup:create REPOSITORIES_STORAGES=gitaly-1 SKIP=tar
    2022-05-10 14:43:52 +1200 -- Backup 1652150628_2022_05_10_15.0.0-pre is done.
  5. Check that just that one repo was backed up. Restore works effectively the same. Create a full backup, and you can restore a specific repo.
    $ tree tmp/backups/repositories/
    └── @hashed
        └── 6f
            └── 4b
                └── 6f4b6612125fb3a0daecd2799dfd6c9c299424fd920f9b308110a2c1fbd8f443
                    ├── 1652150628_2022_05_10_15.0.0-pre
                    │   ├── 001.bundle
                    │   ├── 001.refs
                    │   └── LATEST
                    └── LATEST
    5 directories, 4 files

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by James Fargher