Skip to content

Avoid heap fragmentation issue

Sebastian Ohlmann requested to merge add_mallopt_call into main

Description

When using aligned memory, certain allocation patterns can lead to heap fragmentation with the malloc implementation from glibc. This means that allocated memory cannot be freed or at least not such that new aligned memory can be allocated because allocating aligned memory needs some overhead. This leads to the code exceeding the available memory, i.e., it is similar to a memory leak, although the memory is not really lost, it just cannot be allocated anymore.

The changes in this commit make the implementation in glibc use mmap instead of malloc for allocating arrays larger than 1 MB. This ensures that large arrays can always be freed, thus avoiding heap fragmentation.

This issue has been known to occur in glibc for a long time for specific sizes of arrays. However, it seems that it occurs in Octopus only for very specific batch sizes.

Checklist

Merge request reports