Memory consumption issues

Thank you for your great work on these libraries!

I’ve noticed that under heavy write operations, the modernc memory allocator consumes a lot of memory without releasing it. I did some debugging and came to this. There are 2 possible types of allocations in sqlite

• Ephemeral Allocations: Used in b-trees and elsewhere.
• Cache-like Allocations: Used for pcache, vdbe, and lookaside, which aren’t freed immediately.

With multiple DB connections and a lot of workload, these allocations mix within shared pages of memory allocator, leading to many mmaps (up to 1000 of 1mb in my case) that persist because even a small amount of not-freed allocs at shared pages prevents unmapping.

I’ve partially mitigated the issue by using SQLITE_CONFIG_PAGECACHE, but this isn’t optimal:

• In my case it’s hard to predict the number of databases and the right amount of pre-allocated cache.
• Insufficient pre-allocation falls back to malloc, reintroducing the issue.
• There’s no way to pre-allocate the vdbe cache.

One idea is to implement a custom pagecache via SQLITE_CONFIG_PCACHE2, but it’s complex and still doesn’t solve the vdbe cache case.

I’d appreciate your thoughts on this.