Memory usage can be very high if repeatedly doing RPC `getblock [some_big_block_hash] verbose=2`
Special thanks to @danger89 for insisting we investigate this issue and reporting the bug.
To reproduce:
- Loop in a script, hitting the RPC every 2 seconds, with
getblock
for blockhash000000000000000001b94fb47448c8726bfedd0ef0c4c4ea83948a76ee0d9c74
with verbose=2 (this is a 32MB block). (If you like, do this against MR !1668 (merged) which eats even more memory for this call, but the issue can happen without that MR as well). - Do this for a few minutes and watch process RSS and Virtual Memory climb and stay high (for me it got as high as 4GB before I hit ctrl-C).
- What's worse, the process doesn't appear to ever give back the memory. It stays using 4GB on my Ubuntu 22.04 system for example.
I believe this is a jemalloc memory fragmentation issue. See: https://jemalloc.net/jemalloc.3.html#opt.tcache
- This doesn't happen on macos without jemalloc (macos has a better allocator)
- This DOES happen on Linux with or without jemalloc (Linux uses glibc allocator which is terrible anyway)
- This does not happen on Windows either. It appears to be a Linux x86 jemalloc issue.
The workaround for this would be to start bitcoind with MALLOC_CONF=tcache:false
in the process environment e.g.:
MALLOC_CONF=tcache:false /path/to/bitcoind [args...]
If you do that, then memory fragmentation is drastically reduced and the above test will not lead to memory fragmentation and huge process RSS... at the expense of perhaps some concurrency loss (since the allocator must now acquire a lock for each allocation as multiple threads allocate).