Large backup upload to OpenStack / Swift
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Description
I'm using OpenStack / Swift as the provider to upload my backup files. Unfortunately, my backup has reached 5 GB and hit a limit in the Swift API. As a consequence, backups are not uploaded, and kept local. The retention policy is not applied, and backup are kept forever locally.
The OpenStack Swift API used by Fog to upload backup files requires files larger than 5 GB (the Swift default limit) to be uploaded into segments along with an accompanying manifest file. The fog-openstack documentation shows how to upload such files with chunk and manifest (see link below)
Proposal
Apply the fog-openstack proposal to send the file with multiple chunk along with the manifest.
File to modify : gitlab-rails/lib/backup/manager.rb
Class & method : Manager:upload
Line : if directory.files.create(key: tar_file, body: File.open(tar_file), public: false, multipart_chunk_size: Gitlab.config.backup.upload.multipart_chunk_size, encryption: Gitlab.config.backup.upload.encryption)
My proposal is if provider is openstack & tar_file size exceed the 5GB limit, then apply the fog-openstack code example
Links / references
https://github.com/fog/fog-openstack/blob/master/docs/storage.md#upload_large_files