Skip to content

Add a size-differentiated storage provider

Adam Coldrick requested to merge sotk/cas/size-differentiated-storage into master

Before raising this MR, consider whether the following are required, and complete if so:

  • Unit tests
  • Metrics
  • Documentation update(s)

If not required, please explain in brief why not.

Any relevant metrics are likely more useful in the underlying storages rather than the wrapper.

Description

This is a wrapper around two or more other storages, used to store blobs in a different storage based on their size. This allows reserving a faster, more limited storage such as the LRU in-memory cache for small blobs, falling back to a less constrained storage for large blobs.

Validation

Deploy a BuildGrid using this storage with a small enough size that some blobs will not be in the first storage, for example:

    storages:
      - !lru-storage &lru-storage
        size: 2048M

      - !disk-storage &disk-storage
        path: /home/adam/cas

      - !size-differentiated-storage &cas-storage
        size-limited-storages:
          - max-size: 1K
            storage: *lru-storage
        fallback: *disk-storage

Then run a test build with that BuildGrid.

Issues

Part of #345 (closed) and related to #344 (closed)

Edited by Adam Coldrick

Merge request reports