Add max chunked size for uploading to Object Storage
Proposal
We should let the uploader to accept a specific maximum chunked size to fit a maximum upload size in order to prevent uploading issue.
Rationale
Currently, Cloudflare limits maximum post body size to 100MB for free users, this means uploading to a custom S3 endpoint (such as self-hosted Minio or RADOS) through the free plan of Cloudflare will not be possible if the body size is not chunked to 100MB apiece. It will return a 413 Entity Too Large status code and some of the pipelines cannot proceed due to not being able to upload large artifacts.
As such I recommend to let us control the chunk size just like how we can control multipart_chunk_size_mb
in the Task Runner Backup
Workaround
Disable Cloudflare
Edited by Steve Fan