Skip to content

Add max chunked size for uploading to Object Storage

Proposal

We should let the uploader to accept a specific maximum chunked size to fit a maximum upload size in order to prevent uploading issue.

Rationale

Currently, Cloudflare limits maximum post body size to 100MB for free users, this means uploading to a custom S3 endpoint (such as self-hosted Minio or RADOS) through the free plan of Cloudflare will not be possible if the body size is not chunked to 100MB apiece. It will return a 413 Entity Too Large status code and some of the pipelines cannot proceed due to not being able to upload large artifacts.

As such I recommend to let us control the chunk size just like how we can control multipart_chunk_size_mb in the Task Runner Backup

Workaround

Disable Cloudflare😂

Edited by Steve Fan