feat: Allow configuring parallelism and chunk sizes for cache uploads and...
What does this MR do?
This MR introduces configuration options for parallelism and chunk sizes for cache uploads and downloads, and increases the defaults out of the box. For GoCloud writer (uploads), the change was fairly straightforward because it already exposes the needed options in WriterOptions, but for uploading, the only supported way is using a RangeReader, which involved adding a bit more code. This includes a "detection" mechanism to ensure that the backend supports Ranged GETs, and uses a fallback to the old behaviour if not.
Added docs for the new configuration options and some additional descriptions around configuration of some other semi-related options.
Why was this MR needed?
The performance of cache and artifact handling is currently very poorly suited for jobs with large caches and/or artifacts, i've tested this in a pipeline which was taking around 5 minutes to download a ~5Gb cache, after applying this change (with the included defaults) it dropped to ~20 seconds, similar outcome for artifacts.
What's the best way to test this MR?
Download/Upload some multi-gigabyte cache or artifact files on any CI job