Add fileinfo caching to SiaFile
This MR implements step 3 of #3492 (closed)
It also contains some optimizations I did to speed up the code itself since it seemed like that was actually a bigger bottleneck than disk io. The following changes were made.
-
sfs.mu
is only locked once instead of once per fileinfo returned (got rid of ~7s of lock contention) - multiple threads are used to read from disk (reduced 12s of loading from disk to ~1.2s)
- a third party
Walk
method was used instead offilepath.Walk
(reduced 2-3s of walking over the files to ~0.4s)
On average my roughly 100k files node can now be curled for all files in 2s according to time curl -A "Sia-Agent" localhost:9980/renter/files > /dev/nil
.
Compared to that time ls -Rf > /dev/nil
measures around 6s on my system.
Edited by Christopher Schinnerl