Draft: Health and Bubble Improvements
PROGRAMMING TASK
Description of Task
Improve the performance of the Health loop and bubble code.
Reason or Need for Change
The need for improvement comes from two main issues.
- The Skynet portals are seeing high CPU usage due to the Health Loop and the Bubble code
- The Health Loop code is not able to full update the filesystem within the
healthCheckInterval
which is currently set to 1hr.
Design / Proposal
Health Loop Improvements
Currently the Health Loop is designed to always find the oldest directory and bubble it to update the LastHealthCheckTime
. Additionally the only time the LastHealthCheckTime
is being updated is when the Health Loop calls bubble on that directory. This means that none of the other bubbles that are being called by other systems are iimpacting the LastHealthCheckTime
of the files. The change to the Health Loop being the only place that updates the LastHealthCheckTime
of the files was a recent change to improve the efficiency of bubble.
The ultimate goal for the Health Loop is to keep the AggregateLastHealthCheckTime
within the healthCheckInterval
.
Finding the oldest directory in a very large file system can be a very expensive call. If we can cache or batch some of this process we can save on CPU usage. Any example scenario being where the oldest directories are all at the same level several layers down. The Health loop would go through the expensive searching for those directories to find each one. If we instead tried to update all directories around the oldest directory that needed updating, we could improve the performance of the health loop.
One way to approach this would be to select a size of directory sub tree to update. We could follow the path of oldest directory until we find a directory that is the root directory of a sub tree with 10k files. From there were would update all directories that were outside of the healthCheckInterval
without worrying about which is the oldest since they all need to be updated and we know the oldest is somewhere in the sub tree. Then we call bubble using unique refresh paths to reduce the number of bubble calls.
Bubble Improvements
Bubble is the mechanism used to keep the filesystem metadata up to date. It is used by the Health, Repair, and Stuck loops. There are some controls in place to help protect against excessive bubble calls on a particular directory but there are no global controls that protect against excessive bubble calls filesystem wide.
Existing Controls
There are two existing control mechanisms for the Bubble code.
- Bubble Status
- Unique refresh paths.
Bubble Status allows for us to know if there are any active or pending bubbles being called on a given directory. This information allows us to ignore additional bubble calls for a directory that already has a bubble call pending.
Unique refresh paths allows us to make the least number of bubble calls at a given call site. Since bubbles propagate up through the file system by one bubble call calling bubble on the parent directory when it finishes, unique refresh paths is a helper struct that tracks the lowest level child dirs to call bubble on that will handle updating all the updates needed.
Purposed Controls
Here are two potential controls that could be added to help managed the Bubble code.
- Global Bubble Limiter
- blocking chan with a limit on concurrent bubbles
- only impact places that bubble is called in a blocking manner anyway
- Global unique refresh paths.
- Would be best applied only to individual call sites, not the health loop
- chan that accepts siapaths and bubbles the paths on an interval or one after another in a blocking way.
A Global Bubble Limiter would allow siad to manage the number on concurrent bubbles. This would help address CPU spikes of 200-300%. One easy way to do this would be to use a channel to blocking new incoming bubble requests until there a bubble completed. Since call sites either call bubble in a blocking or non blocking manner already, this should not have a significant impact on the systems that use and call bubble.
A Global unique refresh paths could be integrated into the Bubble Status to additionally only execute a bubble if there is not a lower level child bubble that will trigger it. This would help reduce excessive bubbles on top level directories.