Use bufio to improve CPU usage when writing to disk during downloads. Prior to this MR, a huge portion of the CPU (>60% when using the opimized merkle trees) was going towards making syscalls to write to disk. This MR adds a buffered writer, which substantially reduces the number of syscalls that get made during downloading, freeing up the CPU.
Based on what I can tell, this MR + the merkletree updates from Luke eliminate the CPU as the main bottleneck for downloading in siad, the CPU sits comfortably around 70% (both cores) during a download, which is a big change from the 98% of before. Speeds were improved by 10-15% total between these two optimizations, further speedups will need to focus on other parts of the stack.
Review and complete the checklist to ensure that the MR is complete before assigned to an approver.
- All new methods, or updating methods have clear docstrings
- Testing added or updated for new methods
- Any new packages are added to Makefile and .gitlab-ci.yml
- API documentation updated for API updates
- Module README.md updated for changes to work flow
- Issue added to Sia-UI repo for new supporting features
- Changelog updated