Skip to content

introduce snapshot v5 for faster import

Introduces a new snapshot format thanks to the new irmin snapshot API. It allows to get instantaneous context import using the --no-check option. Otherwise, a consistency check is performed (the overall import time is still drastically improved).

The backward compatibility to import v4 snapshot is maintained. However, it is now only possible to export v5 snapshots.

Benchs

Using a mainnet rolling snapshot at level 2_970_345.

master:

  • import v4 (on-disk): 36min44
  • import v4 (in-memory): 27min37
  • export v4: 12min37

v5 snapshots:

  • import v4: same as import v4
  • import v5-no-check: 15s (110x faster)
  • import v5-default: 12min15 (3x(disk)/2.3x(memory) faster)
  • export v5: 6min30 (2x faster)

In addition to that, the file's size is reduced by 35%: from 6.2Gb to 4.6Gb

Manually testing the MR

  • import legacy
  • export from legacy import
  • display legacy and v5 snapshot info
  • check ux
  • export a snapshot while the GC is running
  • run a GC when a snapshot is exporting

Checklist

  • Document the interface of any function added or modified (see the coding guidelines)
  • Document any change to the user interface, including configuration parameters (see node configuration)
  • Provide automatic testing (see the testing guide).
  • For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
  • Select suitable reviewers using the Reviewers field below.
  • Select as Assignee the next person who should take action on that MR
Edited by Victor Allombert

Merge request reports