Skip to content

Allow SKIP=remote option to be specified during backup

kakakikikeke requested to merge (removed):feature/skip_remote_parameter into master

What does this MR do and why?

When gitlab_rails['backup_upload_connection'] is specified, it is always uploaded to remote storage. Uploading can be skipped by specifying SKIP=remote option when executing the backup command.

Screenshots or screen recordings

# less /home/git/gitlab/config/gitlab.yml

...
  ## Backup settings
  backup:
    path: "tmp/backups"   # Relative paths are relative to Rails.root (default: tmp/backups/)
    gitaly_backup_path: /home/git/gitaly/_build/bin/gitaly-backup
    # archive_permissions: 0640 # Permissions for the resulting backup.tar file (default: 0600)
    keep_time: 1   # default: 0 (forever) (in seconds)
    # pg_schema: public     # default: nil, it means that all schemas will be backed up
    upload:
      # Fog storage connection settings, see http://fog.io/storage/ .
      connection:
        provider: AWS
        region: us-east-2
        aws_access_key_id: AKIxxx
        aws_secret_access_key: 'xxx'
    #   # The remote 'directory' to store your backups. For S3, this would be the bucket name.
      remote_directory: 'my.bucket'
...

# sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=remote

2022-03-29 15:56:05 +0900 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2022-03-29 15:56:09 +0900 -- done
2022-03-29 15:56:09 +0900 -- Dumping repositories ...
{"command":"create","gl_project_path":"root/hoge","level":"info","msg":"started create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2022-03-29T06:56:09.739Z"}
{"command":"create","gl_project_path":"root/hoge.wiki","level":"info","msg":"started create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2022-03-29T06:56:09.742Z"}
{"command":"create","error":"manager: repository empty: repository skipped","gl_project_path":"root/hoge.wiki","level":"warning","msg":"skipped create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.wiki.git","storage_name":"default","time":"2022-03-29T06:56:09.748Z"}
{"command":"create","gl_project_path":"root/hoge","level":"info","msg":"started create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git","storage_name":"default","time":"2022-03-29T06:56:09.748Z"}
{"command":"create","error":"manager: repository empty: repository skipped","gl_project_path":"root/hoge","level":"warning","msg":"skipped create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","storage_name":"default","time":"2022-03-29T06:56:09.747Z"}
{"command":"create","error":"manager: repository empty: repository skipped","gl_project_path":"root/hoge","level":"warning","msg":"skipped create","relative_path":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.design.git","storage_name":"default","time":"2022-03-29T06:56:09.749Z"}
2022-03-29 15:56:09 +0900 -- done
2022-03-29 15:56:09 +0900 -- Dumping uploads ...
2022-03-29 15:56:09 +0900 -- done
2022-03-29 15:56:09 +0900 -- Dumping builds ...
2022-03-29 15:56:09 +0900 -- done
2022-03-29 15:56:09 +0900 -- Dumping artifacts ...
2022-03-29 15:56:09 +0900 -- done
2022-03-29 15:56:09 +0900 -- Dumping pages ...
2022-03-29 15:56:09 +0900 -- done
2022-03-29 15:56:09 +0900 -- Dumping lfs objects ...
2022-03-29 15:56:09 +0900 -- done
2022-03-29 15:56:09 +0900 -- Dumping terraform states ...
2022-03-29 15:56:09 +0900 -- done
2022-03-29 15:56:09 +0900 -- Dumping container registry images ...
2022-03-29 15:56:09 +0900 -- [DISABLED]
2022-03-29 15:56:09 +0900 -- Dumping packages ...
2022-03-29 15:56:09 +0900 -- done
Creating backup archive: 1648536965_2022_03_29_14.9.1-ee_gitlab_backup.tar ... done
Uploading backup archive to remote storage my.bucket ... skipped
Deleting tmp directories ... done
done
done
done
done
done
done
done
done
Deleting old backups ... done. (1 removed)
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.

How to set up and validate locally

  1. Set up s3 config for gitlab-backup (https://docs.gitlab.com/ee/raketasks/backup_restore.html#using-amazon-s3).
  2. Run gitlab-backup task with SKIP=remote option.
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=remote

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports