Skip to content

Replace mark_n_bytes_as_deleted with delete_n_bytes

Jeremiah Bonney requested to merge jbonney/index-manage-cleanup into master

Before raising this MR, consider whether the following are required, and complete if so:

  • Unit tests
  • Metrics
  • Documentation update(s)

If not required, please explain in brief why not.

Description

Cleanup in it's current state operates on an Index and a Storage, coordinating operations on each of these to do a "three-phase" deletion. This ends up being quite complicated and sub-optimal as it goes behind the Index storages back for deletes. This PR moves much of the logic for doing deletions into the Index storages directly, which can decide how best to coordinate deletions between the metadata and the backing storage.

This is done by replacing mark_n_bytes_as_deleted with delete_n_bytes, which returns the number of bytes deleted and handles all the logic within the index itself. This lets the individual implementations handle specifics or optimizations like premarking blobs or caching windows for the SQLIndex.

This change also enables writing sane bulk_delete methods for Index storages, which now coordinate deletions between the backing storage and the index instead of only deleting from the index.

I've also updated the cleanup compose test in the CI to test both the RedisIndex and SQLIndex to help test this change.

Changes proposed in this merge request:

  • Replace mark_n_bytes_as_deleted with delete_n_bytes
  • Simplify bgd cleanup to only operate on Index storages through delete_n_bytes
  • Rewrite bulk_delete for Index storages to delete from both backing storage and index
  • Update cleanup-compose-test CI job to include RedisIndex as well as SQLIndex

Validation

Cleanup should continue to delete the expected amount of data, but should now work on any Index regardless of the underlying storage

Edited by Jeremiah Bonney

Merge request reports

Loading