registry/storage: Make the DeleteFiles method idempotent for all drivers
Context
This change is part of !24 (merged) and related to #10 (closed).
Problem
In !39 (merged), one of the performance bottlenecks we found was a Stat
call being executed before attempting to delete each tag during garbage collection. This happens on vacuum.RemoveManifest
, which is going to be replaced by vacuum.RemoveManifests
once !24 (merged) is merged.
During the implementation of !39 (merged), it was not completely clear the benefit and the impact of this request. Therefore we were uncertain on whether we could avoid it or not, so it remained in the source code.
Upon further investigation, in #16 (comment 273295567), we found that we can't securely bypass Stat
for all storage drivers. Considering the significant performance differences between doing and not doing it, at least for S3 (please see the benchmarks in !39 (merged)) we need a workaround.
Solution
The solution proposed in this MR makes the DeleteFiles
method idempotent for all storage drivers. Therefore, a success will be returned instead of an error if trying to delete an already deleted (or non-existing) object.
By doing this, we can bypass Stat
requests before attempting to remove an object during garbage collection, as a Stat
is usually more costly than a Delete
for a non-existing file (especially if leveraging on bulk requests, like we're doing for S3).
Note: As we don't have integrated tests set up for the Azure, GCS and OSS storage drivers at the moment, I have manually run the TestDeleteFiles*
tests against a real bucket on all these services with success.
Closes #16 (closed)