Skip to content

Add fileinfo caching to SiaFile

Christopher Schinnerl requested to merge siafile-metadata-loading into master

This MR implements step 3 of #3492 (closed)

It also contains some optimizations I did to speed up the code itself since it seemed like that was actually a bigger bottleneck than disk io. The following changes were made.

  • sfs.mu is only locked once instead of once per fileinfo returned (got rid of ~7s of lock contention)
  • multiple threads are used to read from disk (reduced 12s of loading from disk to ~1.2s)
  • a third party Walk method was used instead of filepath.Walk (reduced 2-3s of walking over the files to ~0.4s)

On average my roughly 100k files node can now be curled for all files in 2s according to time curl -A "Sia-Agent" localhost:9980/renter/files > /dev/nil. Compared to that time ls -Rf > /dev/nil measures around 6s on my system.

Edited by Christopher Schinnerl

Merge request reports