Skip to content

Fix Redis Sentinel bug when getting a cache hit with missing blobs

Before raising this MR, consider whether the following are required, and complete if so:

  • Unit tests
  • Metrics
  • Documentation update(s)

If not required, please explain in brief why not.

Description

This MR fixes a bug where an ActionCache hit with missing blobs in CAS would cause an error when using Redis Sentinel. It also adds a docker-compose example using Redis Sentinel to make testing easier.

Validation

Run the Redis Sentinel example.

docker-compose -f docker-compose-examples/redis-sentinel-action-cache.yaml up --scale redis-sentinel=3 --build

Upload an ActionResult that references non-existent blobs, and then try to retrieve it using GetActionResult. The response should be NOT_FOUND with no exceptions on the server side.

A patch to make the bgd actioncache update command upload such an ActionResult:

diff --git a/buildgrid/_app/commands/cmd_actioncache.py b/buildgrid/_app/commands/cmd_actioncache.py
index cc423e9d..c67c71f7 100644
--- a/buildgrid/_app/commands/cmd_actioncache.py
+++ b/buildgrid/_app/commands/cmd_actioncache.py
@@ -22,6 +22,7 @@ from buildgrid.client.channel import setup_channel
 from buildgrid.client.cas import download
 from buildgrid._exceptions import InvalidArgumentError
 from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_pb2
+from buildgrid.settings import HASH
 from buildgrid.utils import create_digest, parse_digest
 
 from ..cli import pass_context
@@ -126,13 +127,20 @@ def update(context, action_digest_string, action_result_digest_string):
 
     # We have to download the ActionResult message from CAS first...
     with download(context.channel, instance=context.instance_name) as downloader:
-        try:
-            action_result = downloader.get_message(action_result_digest,
-                                                   remote_execution_pb2.ActionResult())
-        except Exception as e:
-            click.echo(f'Error: Fetching ActionResult from CAS: {e}',
-                       err=True)
-            sys.exit(-1)
+        # try:
+        #     action_result = downloader.get_message(action_result_digest,
+        #                                            remote_execution_pb2.ActionResult())
+        # except Exception as e:
+        #     click.echo(f'Error: Fetching ActionResult from CAS: {e}',
+        #                err=True)
+        #     sys.exit(-1)
+
+        action_result = remote_execution_pb2.ActionResult(
+            stdout_digest=remote_execution_pb2.Digest(
+                hash=HASH(b"hello world").hexdigest(),
+                size_bytes=11
+            )
+        )
 
         # And only then we can update the action cache for the given digest:
         with query(context.channel, instance=context.instance_name) as action_cache:
Edited by Adam Coldrick

Merge request reports