Restrict size and only add worst chunks to UploadHeap
MERGE REQUEST
Changes to Work Flow
To limit the uploadHeap from growing too large and filling up with non-critical chunks, we will now only add chunks that are less healthy than the directoryHeap and we will limit the size of the uploadHeap to maxUploadHeapChunks.
To achieve this, we are building a temporary heap with the chunks pulled from the files in the directory. By building a temporary heap we also ensure that we are always adding the worst health chunks first. While we build this temp heap we track the worstIgnoredHealth which is the worst health of any file or chunk that we skipped over. worstIgnoredHealth is then used in conjunction with any remaining chunks not added to the uploadHeap to add the directory back to the directoryHeap with an accurately updated health.
Since we will now create situations where not all the chunks that need repairing in a directory are added to the uploadHeap, we will add directories back to the directoryHeap with the health of the chunk that would have been added next. This will make sure that the directory gets fully repaired as well as helping to address cases where a directory only has a few critical chunks and a lot of healthier chunks as it will be re-prioritized in the directoryHeap.
The temporary heap is also limited in size to twice maxUploadHeapChunks so that for large files and directories. If we reach out size limit for the temp heap we will remove the worst half and clear the rest so that we can continue building chunks and ensure that we will always have enough of the worst chunks to fill the uploadHeap.
For the StuckLoop there are no changes as stuck chunks are chosen at random and will always be added to the uploadHeap.
Additionally since we now can get the health of the directory heap, which is the health of the file system, directly, I refactored threadedUploadAndRepair to be Check if healthy -> try and add chunks vs Try and add chunks -> check if healthy.
Testing Updates
Added TestAddDirectoryBackToHeap checks that the directory is added back to the directory heap as expected. Extended TestDirectoryHeap to probe the new managedHealth method of the directory heap.
Modules and Packages Impacted
modules/renter
Working on #3491 (closed)