Skip to content
GitLab Next
  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • tezos tezos
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 1,209
    • Issues 1,209
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 175
    • Merge requests 175
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Tezos
  • tezostezos
  • Merge requests
  • !4862

Improve memory usage of snapshot import/export

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Ioana Cristescu requested to merge icristescu/tezos:snapshot_expose into master Mar 28, 2022
  • Overview 16
  • Commits 8
  • Pipelines 20
  • Changes 25

After the activation of Hangzhou, the memory usage during snapshot import and export increased significantly, reaching 8GB (see issue #2312 and !4076 (merged) for the temporary fix). This is because the very large directories in the context (containing a few million entries) are loaded in memory during snapshot import/export.

This MR exposes the internal nodes used in Irmin to represent these large directories in an optimised way. This considerably reduces the memory usage of the import and export: in our benchmarks, during import, maxrss goes from 7.5GB to 1.6GB while improving the total time as well (see more details here).

This however requires a change to the snapshot format. The import will be backward compatible, however the export will only use this new format.

Requires !4860 (merged).

Manually testing the MR

  • Importing an (old) v2 snapshot, exporting it to test the export.
  • Importing the new snapshot (v3) to test the import.

Checklist

  • Document the interface of any function added or modified (see the coding guidelines)
  • For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
  • Select suitable reviewers using the Reviewers field below.
  • Select as Assignee the next person who should take action on that MR
Edited Apr 14, 2022 by Ioana Cristescu
Assignee
Assign to
Reviewer
Request review from
Time tracking
Source branch: snapshot_expose