Skip to content

Support hash_prefix_size and fix path_prefix support

Jeremiah Bonney requested to merge jbonney/janitor-hash-prefix into master

Before raising this MR, consider whether the following are required, and complete if so:

  • Unit tests
  • Metrics
  • Documentation update(s)

If not required, please explain in brief why not.

Description

This PR fixes adds support for hash_prefix_size, which is used to store blobs in S3 under a hash bucket structure comprised of some number of digits. It does this by generating all the possible prefixes and then shuffles them similarly to the buckets matched by bucket_regex to reduce the chance of multiple janitors repeating work.

This MR also fixes the path_prefix support and adds to both the unit tests and to docker-compose-examples/redis-index.yaml to test it along with hash_prefix_size.

Changes proposed in this merge request:

  • Fix support for path_prefix
  • Add support for hash_prefix_size and shuffle the generated prefixes to distribute load

Validation

You can run the unit test or for a more in-depth test you can use the docker-compose-examples/s3-cas.yaml and docker-compose-examples/redis-index.yaml compose setups.

s3-cas.yaml is the same as before with no prefix or hash prefix, while redis-index.yaml uses both a prefix and a hash_prefix_size of 2, resulting in storing data in S3 like this:

digests/de/adbeef...

The testing instructions from !1030 (merged) can be used to verify the janitor cleans up as expected.

Merge request reports