introduce snapshot v5 for faster import
Introduces a new snapshot format thanks to the new irmin snapshot API. It allows to get instantaneous context import using the --no-check
option. Otherwise, a consistency check is performed (the overall import time is still drastically improved).
The backward compatibility to import v4 snapshot is maintained. However, it is now only possible to export v5 snapshots.
Benchs
Using a mainnet rolling snapshot at level 2_970_345
.
master:
- import v4 (on-disk): 36min44
- import v4 (in-memory): 27min37
- export v4: 12min37
v5 snapshots:
- import v4: same as import v4
- import v5-no-check: 15s (110x faster)
- import v5-default: 12min15 (3x(disk)/2.3x(memory) faster)
- export v5: 6min30 (2x faster)
In addition to that, the file's size is reduced by 35%: from 6.2Gb to 4.6Gb
Manually testing the MR
-
import legacy -
export from legacy import -
display legacy and v5 snapshot info -
check ux -
export a snapshot while the GC is running -
run a GC when a snapshot is exporting
Checklist
-
Document the interface of any function added or modified (see the coding guidelines) -
Document any change to the user interface, including configuration parameters (see node configuration) -
Provide automatic testing (see the testing guide). -
For new features and bug fixes, add an item in the appropriate changelog ( docs/protocols/alpha.rst
for the protocol and the environment,CHANGES.rst
at the root of the repository for everything else). -
Select suitable reviewers using the Reviewers
field below. -
Select as Assignee
the next person who should take action on that MR