Restrict size and only add worst chunks to UploadHeap
Changes to Work Flow
To limit the
uploadHeap from growing too large and filling up with non-critical chunks, we will now only add chunks that are less healthy than the
directoryHeap and we will limit the size of the
To achieve this, we are building a temporary heap with the chunks pulled from the files in the directory. By building a temporary heap we also ensure that we are always adding the worst health chunks first. While we build this temp heap we track the
worstIgnoredHealth which is the worst health of any file or chunk that we skipped over.
worstIgnoredHealth is then used in conjunction with any remaining chunks not added to the
uploadHeap to add the directory back to the
directoryHeap with an accurately updated health.
Since we will now create situations where not all the chunks that need repairing in a directory are added to the
uploadHeap, we will add directories back to the
directoryHeap with the health of the chunk that would have been added next. This will make sure that the directory gets fully repaired as well as helping to address cases where a directory only has a few critical chunks and a lot of healthier chunks as it will be re-prioritized in the
The temporary heap is also limited in size to twice
maxUploadHeapChunks so that for large files and directories. If we reach out size limit for the temp heap we will remove the worst half and clear the rest so that we can continue building chunks and ensure that we will always have enough of the worst chunks to fill the
StuckLoop there are no changes as stuck chunks are chosen at random and will always be added to the
Additionally since we now can get the health of the directory heap, which is the health of the file system, directly, I refactored
threadedUploadAndRepair to be
Check if healthy -> try and add chunks vs
Try and add chunks -> check if healthy.
TestAddDirectoryBackToHeap checks that the directory is added back to the directory heap as expected. Extended
TestDirectoryHeap to probe the new
managedHealth method of the directory heap.
Modules and Packages Impacted
Working on #3491 (closed)