Feature request: Add support for s5cmd using --s3tool
I would like the ability to instead run s5cmd as the --s3tool
for submitting and retrieving blobs from s3 storage.
For uploads, s5cmd is 32x faster than s3cmd and 12x faster than aws-cli. For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s), whereas s3cmd and aws-cli can only reach 85 MB/s and 375 MB/s respectively.
We have a fairly large Gitlab installation running on kubernetes and it is taking many hours to upload to s3 using s3cmd
. That is a very long time to trust that kubernetes will not evict the pod midway during a backup (I know there are various ways to tell the cluster not to evict the pod but with autoscaling on spot instances to save $$$, our nodes can disappear arbitrarily).
Performance with s5cmd
needs to be tested of course but should be much better for this purpose.