Support hash_prefix_size and fix path_prefix support
Before raising this MR, consider whether the following are required, and complete if so:
-
Unit tests -
Metrics -
Documentation update(s)
If not required, please explain in brief why not.
Description
This PR fixes adds support for hash_prefix_size
, which is used to store blobs in S3 under a hash bucket structure comprised of some number of digits. It does this by generating all the possible prefixes and then shuffles them similarly to the buckets matched by bucket_regex
to reduce the chance of multiple janitors repeating work.
This MR also fixes the path_prefix
support and adds to both the unit tests and to docker-compose-examples/redis-index.yaml
to test it along with hash_prefix_size
.
Changes proposed in this merge request:
- Fix support for
path_prefix
- Add support for
hash_prefix_size
and shuffle the generated prefixes to distribute load
Validation
You can run the unit test or for a more in-depth test you can use the docker-compose-examples/s3-cas.yaml
and docker-compose-examples/redis-index.yaml
compose setups.
s3-cas.yaml
is the same as before with no prefix or hash prefix, while redis-index.yaml
uses both a prefix and a hash_prefix_size
of 2, resulting in storing data in S3 like this:
digests/de/adbeef...
The testing instructions from !1030 (merged) can be used to verify the janitor cleans up as expected.