Evaluate performance change in merklize between keeping all blobs in memory vs potentially reading twice
!284 (merged) switched away from using the specialized merklize
functionality of recc
to the version in buildbox-common
. The main difference is that blobs are no longer kept in memory, which means while recc
will use less memory some files may be read twice. While we intuitively think this is worthwhile tradeoff, both for the code de-duplication as well as the savings in memory usage, I'm opening up this issue to provide some concrete numbers for anyone who is interested. If it ends up being a net-negative we can discuss mitigations, and if not at least we'll have some interesting data.