Skip to content

Add ReplicatedStorage to store data in multiple backing storages

Jeremiah Bonney requested to merge jbonney/replicated-storage into master

Before raising this MR, consider whether the following are required, and complete if so:

  • Unit tests
  • Metrics

If not required, please explain in brief why not.

Description

This PR (which superseeds !1038 (closed)) adds the ReplicatedStorage CAS backend, which writes data to multiple storages replicates on reads to keep data in sync. Writes go to all storages, and reads will attempt to replicate data which exists in some storage but not others. This storage follows the philosophy of "If a result can be served successfully return successfully" when it comes to writes/replication, which means the following:

  • Writes are considered successful if at least one storage is written to
  • Reads are successful as long as the data exists in at least one storage

To keep track of potential drift metrics are published for failed writes/replications, and FindMissingBlobs reports the number of blobs which need replication to give visibility into how different the storages are for referenced blobs.

To make the implementation of all this easier I've added a HashableDigest dataclass which allows using set operations on digests in a way that the original Digest protobuf message can be reconstructed easily. This simplifies FindMissingBlobs quite a bit and I think is generally useful.

Changes proposed in this merge request:

  • Add HashableDigest dataclass
  • Add ReplicatedStorage CAS backend

Validation

Added to the standard storage tests as well as some dedicated tests to verify that the replicating works for multiple storages. This can also be validated manually using a config with N disk storages, which lets you easily play around with deleting some/all of the blobs and verifying that data is replicated properly on reads.

Merge request reports