Skip to content

Rollup node: split context irmin suffix on commitment

Alain Mebsout requested to merge alain@functori@split-context into master

Context

Splitting the Irmin context suffix file seems to be necessary to take full advantage of the GC of Irmin (see https://github.com/mirage/irmin/blob/main/doc/irmin-pack/design/chunked_suffix.md). This MR splits the context for each commit which corresponds to a smart rollup commitment because every commitment is a potential GC target. This has the effect of chunking the suffix file which means that instead of having a multi-gigabyte file for the context, there will now be around 1300 files (each file seems to hover around 1MB for the current ghostnet etherlink) containing chunks of the suffix and a GC operation consists in computing the prefix (i.e. the squash until the target commitment) and removing chunk files.

Experiments show that this has little effect on the GC time but greatly reduces the disk space occupied by the context.

Example

mode total GC time context size
archive 0s 8.1MB
full, no split 11.680s 7.0MB
full, with split 11.673s 3.2M

Note

Having ~1000 files for the context should not be an issue (unless the machine is low on available inodes for another reason). It is possible to split the context less frequently though, even if we keep the existing chunks so this could be refined in the future if necessary.

Edited by Alain Mebsout

Merge request reports