Lazy class load increases memory fragmentation
It seems that our preloader of application gems does not preload all parts that are lazily loaded later and in fact make our memory pages defragmented.
For example I tested export
, and export
in some configurations (with Object Storage) uploads archive to the bucket.
This results in loading a number of dependent gems, as they were not used yet: https://gitlab.com/gitlab-org/gitlab-ce/issues/35389#note_211233451
This effectively results in worse memory utilisation as GC of Ruby is not compacting. Thus, I noticed a number of small objects (a few) occupying memory page, and rest being available.
We should consider ensuring that all gems are preloaded into the process before forking, this results in much better memory utilisation, and reduces RSS as GC can much easier unlink the whole pages to the system.
It is likely that loading everything should consume around 20-30MB more, but will reduce memory usage later.
I used heapy
to investigate memory generations when and by whom objects were allocated.