Skip to content

registry/storage: Make the DeleteFiles method idempotent for all drivers

Context

This change is part of !24 (merged) and related to #10 (closed).

Problem

In !39 (merged), one of the performance bottlenecks we found was a Stat call being executed before attempting to delete each tag during garbage collection. This happens on vacuum.RemoveManifest, which is going to be replaced by vacuum.RemoveManifests once !24 (merged) is merged.

During the implementation of !39 (merged), it was not completely clear the benefit and the impact of this request. Therefore we were uncertain on whether we could avoid it or not, so it remained in the source code.

Upon further investigation, in #16 (comment 273295567), we found that we can't securely bypass Stat for all storage drivers. Considering the significant performance differences between doing and not doing it, at least for S3 (please see the benchmarks in !39 (merged)) we need a workaround.

Solution

The solution proposed in this MR makes the DeleteFiles method idempotent for all storage drivers. Therefore, a success will be returned instead of an error if trying to delete an already deleted (or non-existing) object.

By doing this, we can bypass Stat requests before attempting to remove an object during garbage collection, as a Stat is usually more costly than a Delete for a non-existing file (especially if leveraging on bulk requests, like we're doing for S3).

Note: As we don't have integrated tests set up for the Azure, GCS and OSS storage drivers at the moment, I have manually run the TestDeleteFiles* tests against a real bucket on all these services with success.

Closes #16 (closed)

Edited by João Pereira

Merge request reports