Skip to content

Stop overriding fog-aws copy chunk size

Marat Radchenko requested to merge slonopotamus/gitlab:less-http-requests-for-s3-copy into master

What does this MR do and why?

fog-aws >= 3.11.0 now has options to configure upload and copy chunk sizes. This also includes possibility to completely disable chunking (by setting chunk size to non-positive value), which is preferrable for non-Amazon S3 providers like Ceph.

Starting with the same commit, fog-aws automatically enables chunking if file is larger than max_copy_chunk_size.

Ceph doesn't have 5GB chunk limit and shows better performance when copy is done in a single HTTP request.

This commit unblocks #296772. Earlier, the only way to disable chunking was via disabling s3_multithreaded_uploads. Actually it is kinda surprising that s3_multithreaded_uploads has such side effect. See #296772 (comment 751715822) for detailed explanation why chunking needs to be disabled on Ceph currently.

Now, we have an explicit way to disable chunking by setting max_copy_chunk_size to non-positive value.

This commit also documents the way to configure max_copy_chunk_size via GitLab config.

Note that this merge request doesn't mean #285597 (closed) is no longer an issue.

Also see:

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Marat Radchenko

Merge request reports