Implement batch deletion for object storage (artifact) cleanup

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Close this issue

Implement batch deletion for object storage cleanup

Description

Use batch deletion to speed artifact deletion. (we see degraded performance under high load)

Currently, object deletion in our artifact storage system is performed one object at a time. The fog-aws library supports batch deletion through delete_multiple_objects, which would improve performance and efficiency of our cleanup processes for artifact deletion.

Current Status

Single object deletion is used across all storage providers
fog-aws supports batch deletion - https://github.com/search?q=repo%3Afog%2Ffog-google%20delete_multiple_objects&type=code
fog-google does not currently support batch deletion https://github.com/fog/fog-aws/blob/master/lib/fog/aws/requests/storage/delete_multiple_objects.rb
Both underlying http apis from google and aws support batch deletion
GitLab.com's specific object storage configuration needs verification - (via helm chart?)

Proposed Solution

Implement batch deletion using delete_multiple_objects where supported (AWS/S3-compatible providers)
Maintain fallback to single object deletion for unsupported providers
Consider contributing batch deletion support to fog-google, but we may need to fall back on single delete given the # of object storage providers supported.

Technical Considerations

Assess whether direct interface with storage provider is preferable or could unblock this vs contributing to fog
Ensure compatibility with all supported object storage providers (S3, Google Cloud Storage) https://docs.gitlab.com/administration/object_storage/#supported-object-storage-providers

Next Steps

Verify GitLab.com's object storage configuration
Implement conditional batch deletion for supported providers
Create separate issue for contributing batch deletion support to fog-google