fog-aws does not support 5+ GB object storage files and takes too long to copy large files
While testing gitlab-workhorse!466 (diffs), I attempted to upload a 4 GB file via object storage.
The upload finished, but Repositories::LfsStorageController#upload_finalize
timed out after 60 seconds, causing Unicorn to be killed and the client to retry again:
Started PUT "/root/test.git/gitlab-lfs/objects/3bb0163133897ab123927af0b331804c41379c868716db8bd5521ccc963d0d71/4253089792" for 172.30.0.215 at 2020-05-04 05:52:04 +0000
Processing by Repositories::LfsStorageController#upload_finalize as HTML
Parameters: {"file.etag"=>"", "file.md5"=>"93f149a69305b5abdfdf39183f83fd99", "file.name"=>"3bb0163133897ab123927af0b331804c41379c868716db8bd5521ccc963d0d71", "file.path"=>"", "file.remote_id"=>"1588571465-14870-0001-5920-9b99627626dd89d2e9f00b4e15112f1a", "file.remote_url"=>"https://stanhu-testing.s3-us-west-2.amazonaws.com/tmp/uploads/1588571465-14870-0001-5920-9b99627626dd89d2e9f00b4e15112f1a?X-Amz-Expires=15300&X-Amz-Date=20200504T055105Z&X-Amz-Security-Token=redacted&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=REDACTED%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=85559f3140ac38b2bef0b90299a0c30467f1dc9ba792c1a808c8e036739bfffe", "file.sha1"=>"fb9000c4ce32d5fcb40b035f76e3ac4a2023b69c", "file.sha256"=>"3bb0163133897ab123927af0b331804c41379c868716db8bd5521ccc963d0d71", "file.sha512"=>"bb15918568f1cf1f6dbf94ed57b393fed04474a174a40c7cf6f2dc9c17a7f9abce2016a6be97a6ae462347e0b8ce6be869894acd89ac2f009b4125cd78881316", "file.size"=>"4253089792", "namespace_id"=>"root", "repository_id"=>"test.git", "oid"=>"3bb0163133897ab123927af0b331804c41379c868716db8bd5521ccc963d0d71", "size"=>"4253089792"}
My guess is that CarrierWave copying the file from one S3 location to another took longer than 60 s. It seems like we either need to find a way to make this faster, or we have Workhorse handle the finalize step somehow.
This is similar but different from gitlab-foss#31871 (closed), which mostly dealt with NGINX proxying buffering and local directory access, where a mv
is fast if the temporary and final destinations are on the same filesystem.
Edited by Stan Hu