This project is archived. Its data is read-only.

FuseStager: Optimize capturing outputs

Capturing directories is a lot less efficient than it could be right now if they contain many or large unmodified files as we have to read and calculate the hash for each file. This is a common case with BuildStream clients (BuildStream/buildstream#1223 (closed) will improve it but large source trees and integration commands will still trigger this).

With FuseStager I see two possible approaches:

  • As buildbox-fuse keeps track of changes, let it take care of generating the output tree. We'll have to define the interaction between buildbox-run, buildbox-casd, and buildbox-fuse for the capture operation. This depends on buildbox-fuse#32 (moved).
  • Use the user.checksum.sha256 xattr in buildbox-casd. buildbox-fuse already supports this for all unmodified files. I expect this to be less efficient than the other approach but it is probably a lot simpler.

We could start with the latter and implement the former only if we still see it as a bottleneck.

Assignee Loading
Time tracking Loading