High memory allocation in user-defined functions/aggregates arguments
Hi there, first of all thank you so much for this project! It's been a treat to use since switching from the mattn driver.
I help develop a tool that emulates the Google BigQuery SQL Engine by transforming queries into SQLite-compatible queries + functions. As a result, some functions get called many thousands of times per query. When using the Go memory profiler, I see a high number of allocations come from this slice initialization.
Would it be possible to mitigate this by using a sync.Pool for the slice? Do you have any other recommendations on limiting allocations at the driver-level? I could be way off here, but appreciate any thoughts you have
Here's an example profile:
Showing nodes accounting for 3055.28MB, 99.15% of 3081.36MB total
Dropped 339 nodes (cum <= 15.41MB)
flat flat% sum% cum cum%
635.62MB 20.63% 20.63% 637.43MB 20.69% github.com/Recidiviz/go-zetasqlite/internal.DecodeValue
474.79MB 15.41% 36.04% 1116.15MB 36.22% github.com/Recidiviz/go-zetasqlite/internal.setupNormalFuncMap.func1
473.39MB 15.36% 51.40% 473.39MB 15.36% modernc.org/sqlite.errorResultFunction
473.01MB 15.35% 66.75% 473.75MB 15.37% modernc.org/sqlite.funcTrampoline.func1
417.19MB 13.54% 80.29% 684.66MB 22.22% modernc.org/sqlite.functionArgs
315.55MB 10.24% 90.53% 3063.42MB 99.42% modernc.org/sqlite.funcTrampoline
265.73MB 8.62% 99.15% 265.73MB 8.62% modernc.org/libc.GoString
Edited by Dan Hansen