Multipart Upload support should have a deadline per chunk, not one overall deadline
The following discussion from !257 (merged) should be addressed:
-
@nick.thomas started a discussion: (+3 comments) It's a bit unusual (to me, anyway) to have a single very long deadline in which we do lots of faster/smaller requests.
Maybe for a follow-up, but WDYT to the general idea of allocating a time budget per request and so having individual shorter deadlines here?
I'm imagining the case where there are maybe 10 parts, and the second gets stuck through some TCP weirdness. What's the value in allowing it to be stuck for the whole deadline time? The whole request is guaranteed to fail in that case. Much better to give 1/10th of the total timeout to each request so we fail after 2/10ths of the whole deadline time, right?
This relates specifically to multipart object storage upload, where we have a single overall deadline inside which we try to fit N part uploads and 1 "completion" request. This isn't merged yet, but will be soon.
Since the number of parts is known to be limited, could we just specify a deadline of some minutes (perhaps derived from PartSize) per request, and remove the overall deadline?