S3Utils: Support timeout based on blob size
Before raising this MR, consider whether the following are required, and complete if so:
-
Unit tests -
Metrics -
Documentation update(s)
Description
This MR adds the support to timeout S3 requests based on BLOB size instead of always using the hard coded 120s. This will be helpful for uploading or downloading large blobs especially.
To be backward compatible and abled to handle small blobs, configs *_timeout-min-seconds
are also introduced, with the default value of 120s (previous hard-coded default).
Changes proposed in this merge request:
- In
S3Storage
, support configuring timeout based on blob sizes bys3_*_timeout_seconds_per_kilobyte
- Add
*_timeout-min-seconds
in the case that the previous value is too small
Note
moto==4.1.12
breaks a lot of S3 tests for unknown reasons.
Validation
Unit tests
Edited by Zehao Chen