While testing https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466/diffs, I attempted to upload a 4 GB file via object storage. The upload finished, but `Repositories::LfsStorageController#upload_finalize` timed out after 60 seconds, causing Unicorn to be killed and the client to retry again: ``` Started PUT "/root/test.git/gitlab-lfs/objects/3bb0163133897ab123927af0b331804c41379c868716db8bd5521ccc963d0d71/4253089792" for 172.30.0.215 at 2020-05-04 05:52:04 +0000 Processing by Repositories::LfsStorageController#upload_finalize as HTML Parameters: {"file.etag"=>"", "file.md5"=>"93f149a69305b5abdfdf39183f83fd99", "file.name"=>"3bb0163133897ab123927af0b331804c41379c868716db8bd5521ccc963d0d71", "file.path"=>"", "file.remote_id"=>"1588571465-14870-0001-5920-9b99627626dd89d2e9f00b4e15112f1a", "file.remote_url"=>"https://stanhu-testing.s3-us-west-2.amazonaws.com/tmp/uploads/1588571465-14870-0001-5920-9b99627626dd89d2e9f00b4e15112f1a?X-Amz-Expires=15300&X-Amz-Date=20200504T055105Z&X-Amz-Security-Token=redacted&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=REDACTED%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=85559f3140ac38b2bef0b90299a0c30467f1dc9ba792c1a808c8e036739bfffe", "file.sha1"=>"fb9000c4ce32d5fcb40b035f76e3ac4a2023b69c", "file.sha256"=>"3bb0163133897ab123927af0b331804c41379c868716db8bd5521ccc963d0d71", "file.sha512"=>"bb15918568f1cf1f6dbf94ed57b393fed04474a174a40c7cf6f2dc9c17a7f9abce2016a6be97a6ae462347e0b8ce6be869894acd89ac2f009b4125cd78881316", "file.size"=>"4253089792", "namespace_id"=>"root", "repository_id"=>"test.git", "oid"=>"3bb0163133897ab123927af0b331804c41379c868716db8bd5521ccc963d0d71", "size"=>"4253089792"} ``` My guess is that CarrierWave copying the file from one S3 location to another took longer than 60 s. It seems like we either need to find a way to make this faster, or we have Workhorse handle the finalize step somehow. This is similar but different from https://gitlab.com/gitlab-org/gitlab-foss/-/issues/31871, which mostly dealt with NGINX proxying buffering and **local** directory access, where a `mv` is fast if the temporary and final destinations are on the same filesystem.