Skip to content

S3Utils: Support timeout based on blob size

Zehao Chen requested to merge zchen723/s3-timeout-by-blobsize into master

Before raising this MR, consider whether the following are required, and complete if so:

  • Unit tests
  • Metrics
  • Documentation update(s)

Description

This MR adds the support to timeout S3 requests based on BLOB size instead of always using the hard coded 120s. This will be helpful for uploading or downloading large blobs especially.

To be backward compatible and abled to handle small blobs, configs *_timeout-min-seconds are also introduced, with the default value of 120s (previous hard-coded default).

Changes proposed in this merge request:

  • In S3Storage, support configuring timeout based on blob sizes by s3_*_timeout_seconds_per_kilobyte
  • Add *_timeout-min-seconds in the case that the previous value is too small

Note

moto==4.1.12 breaks a lot of S3 tests for unknown reasons.

Validation

Unit tests

Edited by Zehao Chen

Merge request reports