Skip to content

Add multiple directories to Upload Heap

Matthew Sevey requested to merge 3554-add-more-directories into master

MERGE REQUEST

Changes to Work Flow

Currently the repair code, specifically managedAddChunksToHeap, adds chunks from one directory at a time. For small directories this can cause low through put and low back pressure for the repair processes. This MR adds a maxUploadHeapChunks constant that managedAddChunksToHeap will try and fill to. So if after adding all the chunks from one directory there is still room in the uploadHeap, another directory will be added until the uploadHeap is full. All the directory SiaPaths are tracked so that bubble can be called on all of them after the repairs have occurred.

NOTE: maxUploadHeapChunks is currently only being used as a target to fill the uploadHeap up to, it is not currently being used to cap the size of the uploadHeap. That work will be done in a separate MR.

Testing Updates

Add unit test TestAddChunksToHeap to make sure the correct number of chunks are added to the heap.

Modules and Packages Impacted

  • modules/renter

Working on #3554 (closed)

Edited by Matthew Sevey

Merge request reports