After the activation of Hangzhou, the memory usage during snapshot import and export increased significantly, reaching 8GB (see issue #2312 and !4076 (merged) for the temporary fix). This is because the very large directories in the context (containing a few million entries) are loaded in memory during snapshot import/export.
This MR exposes the internal nodes used in Irmin to represent these large directories in an optimised way. This considerably reduces the memory usage of the import and export: in our benchmarks, during import, maxrss goes from 7.5GB
to 1.6GB
while improving the total time as well (see more details here).
This however requires a change to the snapshot format. The import will be backward compatible, however the export will only use this new format.
Requires !4860 (merged).
Manually testing the MR
- Importing an (old) v2 snapshot, exporting it to test the export.
- Importing the new snapshot (v3) to test the import.
Checklist
-
Document the interface of any function added or modified (see the coding guidelines) -
For new features and bug fixes, add an item in the appropriate changelog ( docs/protocols/alpha.rst
for the protocol and the environment,CHANGES.rst
at the root of the repository for everything else). -
Select suitable reviewers using the Reviewers
field below. -
Select as Assignee
the next person who should take action on that MR