Memory consumption issues
Thank you for your great work on these libraries! I’ve noticed that under heavy write operations, the [modernc memory allocator](https://gitlab.com/cznic/memory) consumes a lot of memory without releasing it. I did some debugging and came to this. There are 2 possible types of allocations in sqlite - Ephemeral Allocations: Used in b-trees and elsewhere. - Cache-like Allocations: Used for pcache, vdbe, and lookaside, which aren’t freed immediately. With multiple DB connections and a lot of workload, these allocations mix within shared pages of memory allocator, leading to many mmaps (up to 1000 of 1mb in my case) that persist because even a small amount of not-freed allocs at shared pages prevents unmapping. I’ve partially mitigated the issue by using `SQLITE_CONFIG_PAGECACHE`, but this isn’t optimal: - In my case it’s hard to predict the number of databases and the right amount of pre-allocated cache. - Insufficient pre-allocation falls back to malloc, reintroducing the issue. - There’s no way to pre-allocate the vdbe cache. One idea is to implement a custom pagecache via `SQLITE_CONFIG_PCACHE2`, but it’s complex and still doesn’t solve the vdbe cache case. I’d appreciate your thoughts on this.
issue