Skip to content

OC-1532 Added support for backing up swift containers.

Boros Gábor requested to merge jbzdak/OC-1532/add-backup into master

Created by: jbzdak

Testing instructions

Installation

I will prepare Ansible scripts that deploy backup functionalities on a production environment, however for manual tests it might be easier to use devstack/IM vagrant VMV. Setting up devstack is somewhat cumbersome, if this turns out to be taking a lot of your time, please wait until I prepare a IM sandbox for this feature.

  1. [Optional step] Take a snapshot, or install atop a fresh VM.
  2. Pull this branch.
  3. Download roles cd deploy && ansible-galaxy install -r requirements.yml.
  4. Install tarsnap on the devstack. To install tarsnap you'll need to run ansible scripts on the VM. First get connection details vagrant ssh-config, cd to deploy folder and ansible-playbook install_tarsnap.yml -i localhost, -e "ansible_port=2222" --ssh-common-args "-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" -vvvv -u vagrant -k. This will ask for password which is vagrant. Minor details (like port) might vary.
  5. Add following to .env:
BACKUP_SWIFT_ENABLED = true

BACKUP_SWIFT_TARGET = /var/cache/swift-data-backup
BACKUP_SWIFT_TARSNAP_CACHE_LOCATION = /var/cache/tarsnap
BACKUP_SWIFT_TARSNAP_KEY_LOCATION = /home/vagrant/opencraft/tarsnap.key
  1. Create /var/cache/tarsnap and /var/cache/swift-data-backup on the VM and make vagrant user an owner.
  2. Download tarsnap key attached to OC-1532, and copy it as tarsnap.key in the IM folder.

Testing.

  1. Log in to the VM and run make manage backup_swift. Apart from testing backups. This will test whether: we can recover errors with tarsnap cache (which is entirely missing). You should get "Got tarsnap error. Trying to fix it by running tarsnap --fsck".
  2. Run make manage backup_swift, make manage backup_swift once again. Now "Got tarsnap error. Trying to fix it by running tarsnap --fsck", but you'll get: Object GET failed: https://storage.bhs1.cloud.ovh.net/v1/AUTH_d63f739f98604cb799e584eebbb6057d/safe-to-delete-once-OC-1532-is-merged/canary?multipart-manifest=get 304 Not Modified. This will be explained shortly.
  3. Test that backup task will be run from cron. Either set time on the VM to 1:09 AM or go to backup_swift/tasks.py and change cron to run this every minute.
  4. Download tarsnap backup and verify the contents. To verify tarsnap backup you'll need to:
    1. List tarsnap archives: tarsnap --keyfile /home/vagrant/opencraft/tarsnap.key --cachedir /var/cache/tarsnap/ --list-archives | sort, this will produce a list of time-stamped archives.
    2. List archive contents: tarsnap --keyfile /home/vagrant/opencraft/tarsnap.key --cachedir /var/cache/tarsnap/ -t -f im-swift-backup-2016-06-03T05:46:47.557570
    3. Download the archive cd /tmp && tarsnap --keyfile /home/vagrant/opencraft/tarsnap.key --cachedir /var/cache/tarsnap/ -x -f im-swift-backup-2016-06-03T05:46:47.557570 (the only difference between this and the previous command is -x instead of -t)
  5. To test snitch I think that it would be easiest to: create new snitch at https://deadmanssnitch.com/ (credentials are in the usual place). Set the url as BACKUP_SWIFT_SNITCH and run command manually --- snitch should register the backup. After that just delete the snitch from the https://deadmanssnitch.com/

Pending issues/TODO/ Authors concerns

  • Need to rebase
  • Need to squash commits
  • Due to some arguable design decisions, python-swiftclient decides to handle 304 HTTP statuses when downloading files as errors, that need to be handled (they are) and that are logged. Since I wanted to conserve bandwidth, and make downloading of files faster I decided to use the skip_identical option for swift --- which computes MD5 for the files, and skips them if MD5 match, or rather --- if they match we get 304, this would produce quite a lot of log errors, that need to be suppressed (or this needs to be solved another way)
  • Need to add ansible scripts that will set up production server with tarsnap.

Merge request reports