S3 Driver: Wrap Calls to S3 in a Backoff Algorithm
Context
The S3 driver does not attempt to retry after certain error status that indicate that a service may become available for a request at a later time, such as 503 errors.
The GCS driver implements this technique, so we should be able to adapt this for S3, though we may need to adjust the specific algorithm to better suite S3: https://gitlab.com/gitlab-org/container-registry/-/blob/master/registry/storage/driver/gcs/gcs.go#L626
Solutions
Implement Limited Wrapper
We're considering this as a possible solution in #343 (closed) it's possible that this approach can work nicely for both issues.
Wrap Calls
This is the approach taken by the GCS driver, each call out to the S3 service is wrapped in a retry function.
S3 Documentation
https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
Concerns
Unlike the GCS driver, the S3 driver is also rate-limited, we'll need to ensure that these two feature interact with each other appropriately and predictably.