Loading
Commits on Source 40
-
cznic authored -
cznic authored
Documents the transpilation-based architecture (lib/, vec/, vfs/), the fragile modernc.org/libc version coupling, Makefile targets, debug build tags, the GitLab-canonical / GitHub-mirror workflow, and the singleton Driver registration model. Co-Authored-By:Claude Opus 4.7 (1M context) <noreply@anthropic.com>
-
cznic authored
-
Closes #219. Mirrors the existing FileControlPersistWAL pattern: takes the schema name, allocates a 4-byte slot via the TLS allocator, invokes Xsqlite3_file_control with the SQLITE_FCNTL_DATA_VERSION opcode, and returns the resulting uint32. The returned data version changes whenever the contents of the database file change, so the typical caller polls it to invalidate application-level caches. Adds a regression test that opens a fresh file-backed database, queries the version, performs a commit on the same connection, performs another commit from a separate connection, and asserts the version moves both times.
-
-
cznic authored
-
Following the invitation in !115 review thread.
-
-
The []driver.Value slice passed to user-defined-function scalar and aggregate callbacks, and to vtab Filter / Insert / Update, was allocated fresh on every invocation. For queries that fan out a UDF over many rows this is the dominant per-row allocation, identified as a hot spot in #226 with profile data showing ~13.5% of allocations attributed to functionArgs. The driver's contract on FunctionImpl.Scalar and AggregateFunction.Step / WindowInverse already states the argument values are not valid past the return of the user function, so the slice itself can be reused safely. Add a sync.Pool of *[]driver.Value and route the five existing call sites through acquireUDFArgs / releaseUDFArgs: - funcTrampoline (scalar UDFs) - stepTrampoline (aggregate Step) - inverseTrampoline (aggregate WindowInverse) - vtabFilterTrampoline (vtab Filter) - vtabUpdateTrampoline (vtab Insert / Update) releaseUDFArgs zeroes each entry before returning the slice so any heap references held in the previous invocation can be reclaimed. Benchmark on 1000-row query with a 3-arg noop scalar UDF (Apple M3, Go 1.25, -benchtime 3s -count 3): name old time/op new time/op delta UDFArgsAllocation-8 213000 ns/op 205000 ns/op -4% name old alloc/op new alloc/op delta UDFArgsAllocation-8 118331 B/op 70376 B/op -40% name old allocs/op new allocs/op delta UDFArgsAllocation-8 6754 allocs/op 5754 allocs/op -15% The 1000 saved allocations per iteration match the expected savings: one slice header per UDF invocation, times 1000 invocations per query. Updates #226.
-
After !114 pools the []driver.Value slice across UDF and vtab callbacks, vtab Cursor.Filter and Updater.Insert/Update share the same "not valid past return" contract as FunctionImpl.Scalar / AggregateFunction.Step / WindowInverse. Document it explicitly so vtab implementations don't silently retain references and corrupt later invocations. Per cznic review on !114.
-
-
cznic authored
Co-Authored-By: -
The fix for #198 made (*conn).usable() return false whenever sqlite3_is_interrupted reports the connection is interrupted, so database/sql discards the connection after a context-cancelled query. For file-backed databases that is fine, the data is on disk and a new connection re-opens the same database. For in-memory databases the connection IS the database, so dropping it loses the entire store - re-introducing the regression originally fixed by !74 (issue #196). Detect at open time whether the database is in-memory by checking the output of sqlite3_db_filename and cache the result on the conn. usable() short-circuits to true for those connections so the post-interrupt one stays in the pool. File-backed connections keep the existing behaviour - they are still reported unusable after an interrupt. Adds TestInMemoryDBSurvivesContextCancel with two subtests: one that exercises the regression (insert rows, run a pre-cancelled query, re-query and assert the rows are still there), and one that asserts the file-backed path is still discarded after an explicit sqlite3_interrupt. Closes #196.
-
Per cznic's review on !116, the previous in-memory subtest used ExecContext with a pre-cancelled context, which short-circuits in stmt.exec before Xsqlite3_interrupt is ever called, so the table-still- present check passed even without the fix. Rewrite the subtest to obtain a *conn via db.Conn().Raw(), call sqlite3.Xsqlite3_interrupt directly, and assert c.usable() returns true (matches the file-backed subtest's shape). Retain the end-to-end QueryRow on the shared in-memory cache as a regression sanity check. Verified locally that the subtest fails when the c.inMemory short- circuit in (*conn).usable() is removed.
-
-
cznic authored
Co-Authored-By: -
Follow-up to !114 (#226). !114 pooled the []driver.Value slice header but explicitly left the per-row TEXT/BLOB body copies in place because a default-on zero-copy path would silently corrupt user code that retains an argument across rows -- undetectable by -race (UDF execution is sequential on one goroutine). This commit adds VolatileArgs bool to FunctionImpl as a strict opt-in. When true: - TEXT arguments are unsafe.String views into the SQLite-owned sqlite3_value_text buffer - BLOB arguments are unsafe.Slice views into the sqlite3_value_blob buffer When false (the default for all existing call sites), behavior is byte-for-byte identical to current master. Plumbing: the flag is captured at registration into small wrapper structs keyed in xFuncs.m / xAggregateFactories.m / xAggregateContext.m, so the hot path is one extra field read rather than a second map lookup. The five trampolines (funcTrampoline, stepTrampoline, inverse...
-
Ian Chechin authored
Per @cznic on !120: 1. The volatile branch of functionArgs returned []byte(nil) for empty BLOB args, while the non-volatile branch returned make([]byte, 0). A user comparing args[i] == nil would see different results depending on the flag, which is orthogonal to the volatility contract. Switch the volatile branch to make([]byte, 0) so the empty-BLOB shape is identical across both modes. 2. Document the within-callback re-entrancy hazard in the VolatileArgs docstring: a nested Query/Exec on the same connection during a volatile-args callback can cause SQLite to reuse the underlying value buffers, so a volatile string/[]byte read before the nested call may alias different bytes after it returns. Rare in practice, but useful to spell out alongside the cross-row retention rule. TestVolatileArgsScalar and TestVolatileArgsAggregate stay green.
-
-
cznic authored
Co-Authored-By: -
cznic authored
Co-Authored-By: -
Ian Chechin authored
Follow-up to !120 (#226). !120 added the FunctionImpl.VolatileArgs opt-in for zero-copy TEXT/BLOB access on scalar and aggregate UDF callbacks but left vtab Filter/Update on the non-volatile path, flagged in the MR description as a candidate for a follow-up "if there's demand". This commit extends the same opt-in to: - Cursor.Filter (xFilter) - Updater.Insert / Updater.Update (xUpdate) The contract on those callbacks already says "the vals/cols slice and its entries are not valid past the return of this method; implementations must copy any value they wish to retain", so the API surface is unchanged for users who do not opt in; only the body-allocation strategy differs when opt-in is set. Opt-in is exposed as a new optional interface in package vtab: type VolatileArgsOpter interface { VolatileArgs() bool } A Module that implements it and returns true gets zero-copy TEXT/BLOB views for every Filter, Insert, and Upd...
-
-
cznic authored
-
cznic authored
Co-Authored-By: -
cznic authored
-
Two thin wrappers around the existing sqlite3_backup_remaining and sqlite3_backup_pagecount C symbols. They expose the underlying backup progress counters that the database/sql layer already keeps but that Go callers cannot currently read without dropping to lib/* directly. The motivation is the standard progress-UI use case for online backups: for { more, err := bck.Step(pagesPerTick) if err != nil { return err } ui.Update(bck.PageCount()-bck.Remaining(), bck.PageCount()) if !more { break } } Without these wrappers a caller has to either skip the progress display or fall back to unsafe per-call SQL queries against pragma_page_count. API shape mirrors !115 (FileControlDataVersion): named after the SQLite C function with the s/sqlite3_// prefix stripped and CamelCase applied, documented inline with a link to the official C API page, and added on the existing public... -
-
cznic authored
Co-Authored-By: -
Ian Chechin authored
(*conn).columnText currently allocates twice per TEXT column per row: once for the make([]byte, len) buffer that receives the SQLite-owned UTF-8 bytes, and once again inside the string(b) conversion that runtime.slicebytetostring performs because the compiler must assume the caller could mutate b. Here b is local to columnText and is never touched again after the copy from the C buffer, so the second copy is redundant. Replacing string(b) with unsafe.String(unsafe.SliceData(b), len) builds the returned string directly on top of b. The string is immutable from Go's perspective, the GC keeps b alive for as long as the string is reachable, and no aliasing is possible because b becomes unreachable as []byte the moment the function returns. The same pattern is already used in sqlite.go (!120) for the volatile-args path. Benchmark on darwin/arm64 (Apple M3), 1000-row SELECT of a single TEXT column, -benchtime=2s, before -> after: Short (16-byte TEXT): 4009 -> 4009 allocs/op (Go runtime already short-circuits string(b) for slices below the inline threshold; no regression either) 52348 -> 52348 B/op 157342 -> 155746 ns/op Medium (256-byte TEXT): 5009 -> 4009 allocs/op (-1000 allocs/op = -1 per row) 548351 -> 292350 B/op (-256 KB/op = the second 256-byte copy) 226863 -> 204730 ns/op (-10%) Long (4096-byte TEXT): 5009 -> 4009 allocs/op (-1000 allocs/op = -1 per row) 8228510 -> 4132413 B/op (-4 MB/op = the second 4 KB copy) 1605640 -> 1135113 ns/op (-29%) The saving scales linearly with TEXT column length, since the eliminated work is exactly one memcpy of the column bytes. No change to (*conn). columnBlob, which already returns its make([]byte, len) buffer directly and pays only one alloc + memcpy per row. TestColumnTextScan exercises the path under -race over the three branches of columnText: empty (short-circuit), short (Go-fast-path) and long (allocating) TEXT, including a multi-byte / emoji payload to confirm UTF-8 is preserved bit-for-bit. Full go test -count=1 ./... stays green.
-
-
cznic authored
Co-Authored-By: -
Ian Chechin authored
The Next() hot path calls (*rows).ColumnTypeDatabaseTypeName(i) for every TEXT column on every row when _texttotime=1, and for every INTEGER column on every row when intToTime is set. Each call ran: return strings.ToUpper(r.c.columnDeclType(r.pstmt, index)) which is one libc.GoString to materialise the C decltype string into Go memory, plus a (cheap, allocation-free for already-uppercase inputs) strings.ToUpper. The declared type of a result column is fixed for the lifetime of a prepared statement, so the libc.GoString cost is paid N_text_cols * N_rows times for nothing. Move the lookup to newRows() and cache the uppercased decltype into a new rows.decltypes []string. ColumnTypeDatabaseTypeName, the Next() DATETIME branch (which goes through it), and ColumnTypeScanType now read from the cache instead of redoing the C round-trip per row. The case-sensitive switch in ColumnTypeScanType is rewritten against the cached uppercase values to drop a per-column strings.ToLower as well. Benchmark (darwin/arm64 Apple M3, _texttotime=1, 1000-row SELECT of all DATETIME columns, -benchtime=2s, before -> after): 1 column: 11010 -> 10012 allocs/op (-1000 = -1 per row, the libc.GoString) 400354 -> 392393 B/op (-8 KB = -8 bytes per row, "DATETIME" string body) 646068 -> 601121 ns/op (-7%) 5 columns: 55014 -> 50020 allocs/op (-5000 = -5 per row, -1 per col per row) 2000499 -> 1960654 B/op (-40 KB, scales linearly with columns) 2992839 -> 2908393 ns/op (-3%) The saving scales 1:1 with N_text_cols * N_rows for queries that hit the time-conversion path. Workloads using _texttotime, _time_format, or _intToTime DSN flags benefit; queries without those flags do not touch ColumnTypeDatabaseTypeName per row and see no behavior change. TestColumnTypeDatabaseTypeNameCache covers a mixed-case CREATE TABLE across all SQLite storage classes (INTEGER / TEXT / BLOB / DATETIME / DATE / BOOLEAN), reads the cache once at result-set start and again inside the Next loop for every row, and asserts the values never drift. The full go test -count=1 ./... suite stays green. -
Ian Chechin authored
Per @cznic on !124: the decltype cache rewrites the lowercase decltype switch in ColumnTypeScanType to a cached-uppercase switch, but the existing TestColumnTypeDatabaseTypeNameCache only exercises the DatabaseTypeName side. Add a table-driven TestColumnTypeScanTypeDecltypeCache that covers every arm of the cached switch: - INTEGER + BOOLEAN (any case) -> bool - INTEGER + DATE/DATETIME/TIME/TIMESTAMP -> time.Time - INTEGER + plain / unrecognised decltype -> int64 - TEXT (default) -> string - TEXT + DATETIME-shaped decltype (no flag) -> string - TEXT + DATE/DATETIME/TIME/TIMESTAMP under _texttotime=1 -> time.Time - TEXT + unrecognised decltype under _texttotime=1 -> string Each case uses a mixed-case declared type to keep the case-folding path covered, and inserts one row before SELECT so sqlite3_column_type sees the actual storage class instead of SQLITE_NULL (which would short- circuit ColumnTypeScanType to reflect.TypeOf(nil)). All 15 sub-cases pass under -race.
-
-
cznic authored
Co-Authored-By: -
Ian Chechin authored
(*conn).parseTime ran on every TEXT-stored DATETIME / DATE / TIMESTAMP column read in Next(). The function tried (*conn).parseTimeString first and then walked parseTimeFormats[0..6] sequentially until time.Parse matched the row's value. For the canonical SQLite TEXT datetime format ("2006-01-02 15:04:05.999999999", index 2) every row paid two failed time.Parse attempts in the warmup, plus the one successful match. Each failed Parse allocates a ParseError, so the per-row cost on a steady 1000-row scan was ~5 allocs per row from the format-search alone. Add a sticky per-column hint cache: - rows.parseFmtIdx []int8, sized once at newRows() to the column count, initialised to -1 (no match recorded). - (*conn).parseTime now takes hintIdx int and returns the index that actually matched (or -1 when parseTimeString matched / all formats failed). It tries hintIdx first if in range, then walks the list skipping the index it just tried. ... -
-
cznic authored
Tighten the parseFmtIdx doc comment: a mixed-format column pays at most one extra format probe (on rows whose matching format precedes the cached index), not just the original fall-through cost. Add the !125 CHANGELOG entry. No code/behavior change. Co-Authored-By:
-
cznic authored