Skip to content

Add a RedisIndex class

Jeremy Devenport requested to merge jdevenport/redis-cas-index into master

A RedisIndex uses a redis cache to track which blobs are stored in an underlying storage. This allows FindMissingBlobs requests to be served using only redis while the blobs themselves can be stored using a larger, slower storage provider.

The current implementation supports enough of the IndexABC protocol for bgd cleanup to be able to mark and delete blobs when a high watermark is exceeded. However, it does not provide the LRU interfaces, instead objects that have not been returned by FMB during the protected are simply selected at random.

The redis keys have an expiry time so that unused blobs will automatically age out. However, this requires another process to listen for those notifications and call delete_blob to actually effect the deletion from the underlying storage (and update the index's notion of the total size used in the storage). That process is not yet implemented so for now the setting is not exposed and default to a "long time".

It is also expected that an external "janitor" process will occasionally scan the underlying storage for blobs that have become orphaned from the index for whatever reason.

Merge request reports