Loading
Commits on Source 8
-
Josh Bleecher Snyder authored
Extend page retention beyond the per-class carve target: a region that would be returned to the OS - a drained shared page whose class already has a carve target, or a page backing a single big allocation - now goes into a pool keyed by region size and is reused by later mappings of that size, avoiding the mmap/munmap round trip and the page faults of first touch. Big-allocation turnover is the main syscall source left after the carve-target retention: workloads that repeatedly build and free large buffers (SQLite's JSON functions, for one) pay a dedicated mmap and munmap per buffer. Two measures make the pool actually hit. Region sizes are collapsed into classes - every request up to pageSize maps a full 64 KiB region, larger requests round up to a power of two on 64-bit hosts - so similar-but-unequal sizes reuse each other's regions; the extra address space is never touched and so costs nothing. And the pool is bounded by 4 MiB plus the high-water mark of the live mapping rather than the current one: at the moment a large transient buffer is freed the live size has already shrunk by that buffer, and it is exactly the region most worth retaining. Pooled regions stay in regs and in the counters; Trim returns them to the OS along with the carve targets, and Close already unmaps everything in regs. TestNoTrimBigPage asserted that big-allocation pages are unmapped at once and becomes TestTrimBigPage, asserting the retention and that Trim releases it; TestFreedPool covers reuse. Replaying speedtest1 through modernc.org/sqlite (benchstat n=10, linux/amd64, github.com/josharian/benchmosql): json 380.1m -> 143.9m -62.1% orm 163.5m -> 126.4m -22.7% cte 487.0m -> 442.0m -9.3% main 384.1m -> 364.7m -5.1% star 26.4m -> 25.2m -4.7% fp 306.0m -> 292.9m -4.3% app 74.3m -> 75.4m +1.4% geomean -18.8% -
Josh Bleecher Snyder authored
Aligning a fresh mapping to a pageSize boundary costs up to two munmap calls for the trimmed-off ends, so a cold allocator paid roughly three syscalls per 64 KiB slab region. When a single pageSize region is needed and the pool is empty, acquire slabBatch of them in one mapping and put the rest into the pool: three syscalls amortized over sixteen regions, and the pooled remainder immediately serves the other size classes, which all share the pageSize region class. Unix only: VirtualFree with MEM_RELEASE frees only whole VirtualAlloc allocations, never a sub-range, so on Windows a carved region could not be returned individually (canCarve). This trims the mapping churn of a cold or recently Trimmed allocator - a process that touches eleven size classes now issues one mmap instead of eleven mmaps and up to twenty-two trim munmaps. Steady-state throughput of long benchmark runs is unchanged: once the pool is warm, fresh mappings are rare either way. TestTrim and TestTrimClasses asserted exactly one mapping per class and now assert stability of the mapping count while cycling, plus the batch-rounded count; TestSlabBatch covers the batch acquisition itself.
-
Josh Bleecher Snyder authored
-
cznic authoredNot for merge as-is. This sizes one tradeoff on top of this branch: the region pool from e8bac908 is bounded by 4 MiB + hiLive, so a process that briefly peaks keeps that peak resident until something calls Trim. Peaking at 2 GiB of 4 KiB allocations and then freeing all of it: VMAs after drain RSS after drain v1.12.0 34 13 MB opt 2,209 2,063 MB opt + this 2,212 158 MB The pool holds two things at once: an address range, which is cheap and is what buys the saved syscalls and the saved map entries, and a resident set, which is expensive and which nobody asked for. madvise separates them. The mapping stays - and with it the address, the VMA and the absence of a future mmap - while the physical pages go back. MADV_DONTNEED does not change vm_flags, so it does not split the VMA: the map counts above are unchanged by it. Two details keep this small rather than a redesign. The first OS page of each region is skipped, so the page header survives and nothing has to be reconstructed when the region comes back out of the pool. And a hot window - the newest hotBytes worth of each size class - stays committed, because those are the regions a churning workload takes straight back, where decommitting only buys a page fault. speedtest1 --size 100 through modernc.org/libsqlite3, four builds interleaved round-robin x7, medians: wall vs v1.12.0 v1.12.0 10.39 s - opt 9.26 s -10.9% opt + decommit on every retain 9.51 s -8.5% opt + decommit, 4 MiB hot window 9.19 s -11.6% The hot window costs nothing measurable against opt while returning the hoard, which is the interesting part: there may be no fast-versus-lean axis here that needs exposing as a knob at all. MEMORY_MADV and MEMORY_HOT_MB are scaffolding so this can be A/B'd against a real workload without rebuilding; neither is proposed API. MADV_DONTNEED is the default rather than MADV_FREE because MADV_FREE reclaims lazily and the 2 GiB above still read as 2 GiB of RSS - the same reason the Go runtime defaults to MADV_DONTNEED. Windows keeps current behaviour; the analogue there is VirtualAlloc(MEM_RESET) or DiscardVirtualMemory. go test -tags=memory.counters passes with the policy on and off, go vet -unsafeptr=false and staticcheck are clean, and all supported targets cross-build. Co-Authored-By:
Claude Opus 5 (1M context) <noreply@anthropic.com>
-
cznic authored
retain() decommitted s[len(s)-1-hot] on every push without tracking what was already decommitted, so a pool oscillating around the boundary re-madvised the same region on every Free. Harmless, but it reintroduced the syscall-per- turnover cost the pool exists to avoid, at exactly the depth a steady-state workload sits at. a.cold[size] now counts the stack's decommitted bottom. The invariant: freed[size][:cold[size]] is decommitted, everything above is committed. retain decommits s[cold] and advances the watermark only when the committed top exceeds the hot window, and the reuse pop clamps the watermark when the pool drains into the prefix. The region decommitted is the same one the old code picked - a decommit fires only when the committed count is hot+1, where s[cold] and s[len(s)-1-hot] coincide - so the policy is unchanged, only the repeats are gone. Oscillating one 64 KiB-class malloc/free against an 80-deep pool (hot window 64 regions), linux/amd64, 100k cycles: madvise calls wall/cycle before 100,018 1.90 us after 18 48 ns go test passes with and without -tags=memory.counters, also under MEMORY_MADV=none, MEMORY_MADV=free and MEMORY_HOT_MB=0; go vet -unsafeptr=false and staticcheck are clean, and all supported targets cross-build. Reported-by: Josh Bleecher Snyder Co-Authored-By: -
cznic authored
MEMORY_MADV and MEMORY_HOT_MB existed so the decommit policy could be gauged against a real workload without rebuilding. That happened: production simulations put the policy at nearly no cost against plain retention while returning the hoard, and the branch was asked to land in this shape. So the policy becomes what 56fccba2 said it would if it survived - MADV_DONTNEED and a 4 MiB hot window as compile-time constants, no environment surface. Anyone still A/B-ing with the env vars: pin b447515e, the last commit that has them. go test passes with and without -tags=memory.counters, go vet -unsafeptr=false and staticcheck are clean, and all supported targets cross-build. Co-Authored-By:
-
cznic authored
The unix decommit policy ends at madvise, so on Windows the pool kept its regions committed. That is worse there than holding RSS is on unix: Windows does not overcommit, so a drained pool holds pagefile-backed commit charge against the machine-global limit for as long as it is retained. VirtualFree(MEM_DECOMMIT) is the analogue - the reservation and with it the address range survive while the pages and the commit charge go back - with one asymmetry: MEM_DECOMMIT'ed pages fault on touch instead of soft-faulting zeroes the way MADV_DONTNEED'ed pages do. So the reuse path grows a recommit step: when a pop comes out of the pool's decommitted prefix - exactly what the cold watermark tracks - the region's tail is recommitted zero-filled via VirtualAlloc(MEM_COMMIT) before it is handed out, and a recommit failure (machine out of commit) leaves the pool untouched and surfaces as the allocation error it is. The same MEM_DECOMMIT/MEM_COMMIT pairing the Go runtime uses in sysUnused/sysUsed on Windows. The platform split becomes: canDecommit gates the policy per platform, decommit/recommit implement it, and hotBytes moves to shared code. decommit_other.go is gone - the package only ever built where mmap_unix.go or mmap_windows.go exists, so the !unix stub covered nothing real. Measured on the windows/amd64 builder (8000 x 64 KiB dedicated regions, peak ~500 MB, then free everything): drained pool: working set commit master (38f7932c) 475.4 MB 514.2 MB this commit 41.4 MB 49.4 MB The residue is the 4 MiB hot window plus the 7936 committed header pages. Re-peaking through the pool recommits all 7936 cold regions with every page written, and Trim's MEM_RELEASE handles partially decommitted allocations. go test passes with and without -tags=memory.counters on windows/amd64 and linux/amd64, the unix decommit microbenchmark is unchanged, go vet -unsafeptr=false and staticcheck are clean, and all supported targets cross-build. Co-Authored-By: -
cznic authored
TestFreedPool failed on windows/386, the first farm run of the pool: a 1 MiB allocation was retained but never reused. mmapSize classed the request by OS pages (1 MiB + 4 KiB) while mmap_windows rounded the actual mapping to VirtualAlloc's 64 KiB granularity (1 MiB + 64 KiB) and page.size recorded that - and page.size is what retain keys the pool by. The lookup missed its own region forever, so such sizes never reused and the pool accumulated regions no request could reach. Only windows/386 could see it: on 64-bit windows the power-of-two class is already a 64 KiB multiple, and the unix mmap returns exactly what was asked, so the class and page.size agree everywhere else - the farm's win64, pi400 and e5-1650 all passed the same commit. mmapSize now rounds classes to mmapGranularity, what the platform mmap in fact maps at: osPageSize on unix, pageSize on windows. The class of a request is then the page.size its mapping comes back with. Latent since e8bac908; unix behavior is bit-for-bit unchanged. go test passes with and without -tags=memory.counters on linux/amd64, go vet -unsafeptr=false and staticcheck are clean, and all supported targets cross-build. Co-Authored-By: