!48269 (merged) enabled LFS
clients to use chunked transfers via
However, in some cases, this breaks uploads to AWS S3 if specific-storage settings are used. It appears the problem is exposed when HTTP/1.1 is used. HTTP/2.0 automatically buffers a chunked transfer.
When direct uploads are used, GitLab will only generate pre-signed,
multipart URLs if the
Content-Length parameter is sent by the
client. Previously when chunked transfers were not supported, this
header was always available and thus was hard-coded in the LFS storage
When this header is no longer available, the Workhorse
attempts to transfer the file with an S3 PutObject API call with
Transfer-Encoding: chunked. This will return a
501 Not Implemented
error since this header is not supported; S3 has a different mechanism
to chunked transfers. Note that Google Cloud Storage works fine in this
Content-Length is not always available, we have a few
- Disable LFS chunked transfers.
- Re-enable request buffering in NGINX.
- Modify Workhorse to tell us whether
- Be pessimistic and always generate multipart URLs and use a max length with the size of file.
Option 1 is not optimal because we still want to support LFS chunked transfers, especially for GitLab.com where Cloudflare will reject files over 5 GB.
Option 2 is not desirable either because it causes NGINX to store a large temporary file and delays the transfer to object storage.
Option 3 is a slightly preferable option, but it involves modifying Workhorse as well. We should consider this as a follow-up issue.
To fix the immediate problem, we implement option 4. Note that using consolidated object storage settings avoids this problem because Workhorse handles the upload natively with the AWS SDK and doesn't need presigned URLs.
Relates to gitlab-workhorse#292 (moved)