Add fileinfo caching to SiaFile
This MR implements step 3 of #3492 (closed)
It also contains some optimizations I did to speed up the code itself since it seemed like that was actually a bigger bottleneck than disk io. The following changes were made.
sfs.muis only locked once instead of once per fileinfo returned (got rid of ~7s of lock contention)
- multiple threads are used to read from disk (reduced 12s of loading from disk to ~1.2s)
- a third party
Walkmethod was used instead of
filepath.Walk(reduced 2-3s of walking over the files to ~0.4s)
On average my roughly 100k files node can now be curled for all files in 2s according to
time curl -A "Sia-Agent" localhost:9980/renter/files > /dev/nil.
Compared to that
time ls -Rf > /dev/nil measures around 6s on my system.