OC-1532 Added support for backing up swift containers.
Created by: jbzdak
Testing instructions
Installation
I will prepare Ansible scripts that deploy backup functionalities on a production environment, however for manual tests it might be easier to use devstack/IM vagrant VMV. Setting up devstack is somewhat cumbersome, if this turns out to be taking a lot of your time, please wait until I prepare a IM sandbox for this feature.
- [Optional step] Take a snapshot, or install atop a fresh VM.
- Pull this branch.
- Download roles
cd deploy && ansible-galaxy install -r requirements.yml
. - Install tarsnap on the devstack. To install tarsnap you'll need to run ansible scripts on the VM. First get connection details
vagrant ssh-config
, cd to deploy folder andansible-playbook install_tarsnap.yml -i localhost, -e "ansible_port=2222" --ssh-common-args "-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" -vvvv -u vagrant -k
. This will ask for password which isvagrant
. Minor details (like port) might vary. - Add following to
.env
:
BACKUP_SWIFT_ENABLED = true
BACKUP_SWIFT_TARGET = /var/cache/swift-data-backup
BACKUP_SWIFT_TARSNAP_CACHE_LOCATION = /var/cache/tarsnap
BACKUP_SWIFT_TARSNAP_KEY_LOCATION = /home/vagrant/opencraft/tarsnap.key
- Create /var/cache/tarsnap and /var/cache/swift-data-backup on the VM and make vagrant user an owner.
- Download tarsnap key attached to OC-1532, and copy it as
tarsnap.key
in the IM folder.
Testing.
- Log in to the VM and run
make manage backup_swift
. Apart from testing backups. This will test whether: we can recover errors with tarsnap cache (which is entirely missing). You should get "Got tarsnap error. Trying to fix it by running tarsnap --fsck". - Run
make manage backup_swift
,make manage backup_swift
once again. Now "Got tarsnap error. Trying to fix it by running tarsnap --fsck", but you'll get:Object GET failed: https://storage.bhs1.cloud.ovh.net/v1/AUTH_d63f739f98604cb799e584eebbb6057d/safe-to-delete-once-OC-1532-is-merged/canary?multipart-manifest=get 304 Not Modified
. This will be explained shortly. - Test that backup task will be run from cron. Either set time on the VM to
1:09 AM
or go tobackup_swift/tasks.py
and change cron to run this every minute. - Download tarsnap backup and verify the contents. To verify tarsnap backup you'll need to:
- List tarsnap archives:
tarsnap --keyfile /home/vagrant/opencraft/tarsnap.key --cachedir /var/cache/tarsnap/ --list-archives | sort
, this will produce a list of time-stamped archives. - List archive contents:
tarsnap --keyfile /home/vagrant/opencraft/tarsnap.key --cachedir /var/cache/tarsnap/ -t -f im-swift-backup-2016-06-03T05:46:47.557570
- Download the archive
cd /tmp && tarsnap --keyfile /home/vagrant/opencraft/tarsnap.key --cachedir /var/cache/tarsnap/ -x -f im-swift-backup-2016-06-03T05:46:47.557570
(the only difference between this and the previous command is-x
instead of-t
)
- List tarsnap archives:
- To test snitch I think that it would be easiest to: create new snitch at https://deadmanssnitch.com/ (credentials are in the usual place). Set the url as
BACKUP_SWIFT_SNITCH
and run command manually --- snitch should register the backup. After that just delete the snitch from the https://deadmanssnitch.com/
Pending issues/TODO/ Authors concerns
-
Need to rebase -
Need to squash commits -
Due to some arguable design decisions, python-swiftclient
decides to handle304
HTTP statuses when downloading files as errors, that need to be handled (they are) and that are logged. Since I wanted to conserve bandwidth, and make downloading of files faster I decided to use theskip_identical
option for swift --- which computesMD5
for the files, and skips them if MD5 match, or rather --- if they match we get 304, this would produce quite a lot of log errors, that need to be suppressed (or this needs to be solved another way) -
Need to add ansible scripts that will set up production server with tarsnap.