Skip to content

Implement deferred writes to the fallback for with-cache

Adam Coldrick requested to merge sotk/cas/with-cache-deferred-writes into master

Before raising this MR, consider whether the following are required, and complete if so:

  • Unit tests
  • Metrics
  • Documentation update(s)

If not required, please explain in brief why not.

I don't immediately see any useful metrics here; we already have metrics for the duration of a write request.

This MR still needs documentation, and could maybe do with benchmarking with real workloads.

Description

Currently, writing to the !with-cache-storage blocks until the blob is written to both the cache layer and the fallback layer. This is desired behaviour in some circumstances, but for some combinations this adds an unnecessary wait.

This MR adds an optional flag which causes the commit_write method to finalise the fallback's write in a background thread, and return once the blob is written into the cache layer. This removes the need for writers to wait for writes to slow storage backends, such as S3.

Obviously, this raises issues in cases where the cache is small or (when using multiple BuildGrids) shared between instances, and so should only be used in particular circumstances. An example use case could be a large Redis cache with an S3 fallback. Writes to S3 are much slower than writes to Redis, and a clustered Redis deployment should be reliable enough to only encounter minimal issues.

Validation

Deploy a with-cache storage with defer-fallback-writes: yes set, and run some test builds.

Issues addressed

Part of #344 (closed)

Edited by Adam Coldrick

Merge request reports