Skip to content

Memory and CPU usage high for large/binary heavy sites

Antora parallelizes beautifully but this does increase CPU usage (I understand that on a headless server this is not a very great issue). However I think it is this as well as the moving of everything (all repositories, all tags) into memory which is causing me difficulty.

My documentation is "binary heavy" and I have a fair number of versions and components. While all of these stored in git only consume about 3.4 Gb of memory, when I carry out my Antora build, sometimes I can crash KDE on my laptop or more often an out of memory situation causes the kernel to kill node.

The processor usage (I have four cores) will sometimes max out at around 90 during a build. It seems to require on the order of 10 Gb of memory (probably because the repositories are compressed). Unfortunately my laptop only has 8 Gb (silly me!) so swapping out while using significant CPU reduces my machine to a crawl.

Before running Antora I first must stop any music (to avoid the jitter), close any code editors and shutdown my browser.

I suspect this is a problem caused by large number of binary artifacts + a large number of components and versions.

I've tried using nice/ionice to reduce the resource usage with somewhat mixed success.

I'm not sure how this should best be resolved but I suspect Antora's handling of binary artifacts could be modified so these weren't brought into memory and that this would (for my use case and any future git-lfs integration, see #185) massively reduce memory consumption and probably increase speed for site generation (I am not sure but I think towards the end of the build there is quite a slow -- possibly even single threaded generation of files in the site generator).

For development I will simply reduce the number of components I use for testing so this issue is by no means urgent (from my point of view) and ultimately I intend to run Antora in headless situation (although I would still rather run it on a cheap server without large numbers of processors or much memory and am happy for generation to go slower...).

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information