Skip to content

Draft: Adjust which storages are enabled for cleanup

Jeremiah Bonney requested to merge jbonney/enable-cleanup-by-default into master

Before raising this MR, consider whether the following are required, and complete if so:

  • Unit tests
  • Metrics
  • Documentation update(s)

If not required, please explain in brief why not.

Description

What storages can be cleaned up by bgd cleanup is controlled by the is_cleanup_enabled method, which defaults to False in the StorageABC. Right now only the S3 storage is marked as being eligible for cleanup, which is very restrictive and prevents using more complicated storage layers under an Index. This PR flips the logic and marks storages as enabled for cleanup by default, requiring storages which can't be cleaned up to opt-out.

It also has special handling for the sharded and size-differentiated storages to check the backing storages to determine if they themselves can report as having cleanup enabled.

NOTE Marking this as draft for now. I'm going to experiment with some more in-depth changes around cleanup which might clean this up a bit.

Edited by Jeremiah Bonney

Merge request reports