Skip to content

Draft: Proof-of-concept: Restart the HTTP RPC worker threads occasionally to remove any thread-local allocation caches

This intends to address #466 using a novel approach of re-starting the HTTP RPC worker threads periodically (only if they are idle and only if at least 60 seconds has elapsed after creation AND they have processed at least 1000 work items).

This killing/re-creation of the worker threads forces the jemalloc "tcache" to be deallocated each time, thus reducing memory usage.

To reproduce the issue seen in #466 simply do, via RPC: getblock 000000000000000001b94fb47448c8726bfedd0ef0c4c4ea83948a76ee0d9c74 2 in a loop a number of times, which will permamnetly grow process RSS to 1G, 2G, or 3G usage permanently.

With this commit the RSS usage eventually falls back down when the RPC activity ceases. Without it, it doesn't.

This is not intended for merge at this time, it is merely a proof-of-concept.

I don't like this implementation in that while it is correct, it is invasive and kind of not as succinct as I would have liked, to be frank. Also I am not sure the added complexity to this subsystem is warranted given the corner-case-nature of #466.

That being said -- it would be safe code to merge into master and to have in our codebase as it is a decent workaround to a know problem with both glibc and jemalloc allocators.

Merge request reports