feat(storage/driver/s3): run DeleteFile batches in a loop instead of spawning goroutines
This MR seeks to reduce some instability in garbage collection sweep stages for S3, as experienced here: #753 (closed)
Instead of spawning goroutines for each batch of deleted files, this MR uses a simple loop.
While this might theoretically slow down the deletion of files, we've seen S3 generate a lot of 503
responses to these operations, even adjusting down maxrequestspersecond
.
Since this is used in the sweep stage, any amount of work accomplished is "kept" between runs.
This is in contrast to the mark stage, which needs to fully complete to generate any value.
Therefore, it's worth going slower here to ensure we can accomplish the most amount of deletion.
Edited by Hayley Swimelar