`Downloader.download_directory()` issues messages larger than the gRPC limit
Context
When downloading directories, the BuildGrid CAS client issues BatchReadRequests
for blobs that exceed the size set by MAX_REQUEST_SIZE
. It then fails printing an error that looks like this one:
Error: Downloading directory: Received message larger than max (6831133 vs. 4194304)
This seems to be caused in the Downloader._write_directory
method, where a file is queued regardless of its size.
Task Description
Describe the steps to take / implementation plan. If it's a bug, do you know how it may be tackled?
-
Make Downloader._write_directory()
check whether the size of the file is in the range where it is allowed to be fetched usingBatchReadBlobs()
and if not fetch it usingByteStream.Read()
Acceptance Criteria
BuildGrid correctly downloads directories with files with that are larger than MAX_REQUEST_SIZE
while batching requests as much as possible.
Edited by Santiago Gil