Skip to content

[SE-122] Management command to migrate static data from SWIFT to S3

Daniel Clemente Laboreo requested to merge clemente/migrate-swift-to-s3 into master

This PR adds a management code to migrate an instance from using SWIFT for files (attachments, forum uploads, exams, …) to using S3. It moves the files (with rclone) and then changes the instance's storage setting. It includes Rocío's code.

It has already been used to migrate many trial instances. It will be used very few more items and doesn't need fancy features. It doesn't have unit tests since it does so much remote communication that we would have to mock rclone and AWS, and it's not worth it for a few-times script. It has been tested with real servers and verified to work there.

The script fixes some of the filename bugs we found in servers (e.g. {submissions_attachmentsuser_tasks/ ---> submissions_attachments/user_tasks). There may be more edge cases but the current fixes cover all of our servers.

To choose the set of servers you want, you need to change some line. You can also easily enable/disable particular sections (useful if you need to retry without creating the user/bucket each time).

To run it: honcho run ./manage.py migrate_swift_to_s3 (from Ocim). Or better, follow the included instructions at the top of the file.

Merge request reports