Add sqlite-vec support

See merge request !93

See CHANGELOG.md for more information

_timezone accepts an IANA timezone name, such as
"UTC", and applies it to both reads and writes.

For writes, time.Time values are converted to the
target timezone before formatting.

For reads, timezone-less strings are interpreted
as being in the target timezone.

Supporting configuration at the DB driver level
helps avoid industrial accidents when working with
SQLite datetime strings, which lack time zones.

So that time.Times may be formatted identically to how SQLite
does it natively with datetime and CURRENT_TIMESTAMP.

add _timezone DSN query parameter

See merge request !94

add _time_format=datetime support

See merge request !95

This particular loop occurs a lot.
Give it a name and make it concise.
No functional changes.
The goal is to make upcoming changes a bit clearer.

The multi-statement path was subject to both double-frees and leaks.

Having allocs declared once and re-used across loop iterations
made it hard to manage correctly. Reduce its scope to within the
inner closure and defer freeAllocs there.

Even within a loop, it was easy to mismanage; force callers to deal
with this consistently by passing a pointer to newRows and zeroing there.

Take care to Close rows objects from previous iterations when
a subsequent statement in a multi-statement query fails.

The tests here demonstrate both double-frees and
leaked memory without these changes.

I noticed while in this code that in the optimized
single-statement path, there are some unnecessary calls to
reset/clearBindings after newRows. When newRows fails, its internal
defer calls r.Close, which finalizes the prepared statement.
The subsequent reset and clearBindings calls would then operate
on an already-finalized handle. This would be bad, except that
columnCount/colum...

improve memory safety of allocs in stmt.query

See merge request !96

The column values and rowid were swapped;
vtabUpdateTrampoline wasn't following SQLite's xUpdate contract.

The existing tests didn't catch this because they used
only a single column. Add more columns, and expand the test.

This is pretty standard Go fare,
but it's worth calling out as a cheap one-liner.

The alternative is to remove this restriction,
but I don't think it's worth the mutex overhead.

fix vtab argv layout

See merge request !97

Josh Bleecher Snyder authored Apr 03, 2026 and cznic committed Apr 03, 2026

iKey2 is an int64, not two int32s.

The trampoline's parameter list appears to have been modeled after the
internal SQLite function sqlite3VdbePreUpdateHook, which takes
(i64 iKey1, int iReg, int iBlobWrite).

But that internal function computes iKey2 from iReg before invoking the
public callback, whose actual signature ends with
(sqlite3_int64 iKey1, sqlite3_int64 iKey2).

The trampoline declared the final slot as (iReg int32, iBlobWrite
int32) instead of (iKey2 int64), which works except for large rowids.
Depending on the arch, this likely manifests as truncation.

It's rather unclear how important this is, but on the other hand,
the fix is small and targeted and highly testable,
and for e.g. snowflake-style IDs, it'd show up as silent corruption.

fix preUpdateHookTrampoline signature

See merge request !98

document that registration must precede Open

See merge request !99

tls.Alloc is inappropriate here, because sqlite
owns this pointer and may choose to resize it;
we must use a sqlite allocator.

The test triggers a crash using a close,
but it is also possible (albeit longer and noisier)
to trigger via a re-alloc/re-size.

fix Deserialize allocator

See merge request !100

How to handle Deserialize(nil) is a judgment call.

sqlite3_deserialize(db, schema, NULL, 0, 0, ...) disconnects and starts fresh.
But in database/sql world, connections are pooled, so that seems bad.
But nil as an input is guaranteed to fail, possibly in ugly ways, if not handled explicitly.

Given that we're in Go, and creating a new DB is trivial,
opt for safety and explicitness and early feedback,
and declare Deserialize(nil) to be an error and handle it explicitly.

reject Deserialize(nil) with clear error

See merge request !101

Josh Bleecher Snyder authored Apr 04, 2026 and cznic committed Apr 05, 2026

Similar to 5e394aa2,
except without practical consequences, and therefore untestable.

Low priority, but trivial.

Fixing only to avoid leaving a code rake.

fix commitHookTrampoline and rollbackHookTrampoline signatures

See merge request !102

Josh Bleecher Snyder authored Apr 04, 2026 and cznic committed Apr 06, 2026

`go test -race .` fails for me without this fix.

The sqlite3_module struct was allocated on the Go heap and passed
to sqlite3_create_module_v2 as a uintptr. Later when we access fields
through unsafe.Pointer arithmetic, Go's checkptr instrumentation
rejects the dereference because the pointer was round-tripped through
a uintptr in C-allocated memory.

Use libc.Xcalloc instead, matching the pattern in mutex.go and the VFS code.

The allocation is intentionally never freed; modules are registered once
and must outlive every connection that references them.
There is no UnregisterModule API.

This is a prerequisite change to another fix in the vtab code
which requires `go test -race` to reproduce.

allocate sqlite3_module with C allocator to fix checkptr

See merge request !103

Josh Bleecher Snyder authored Apr 05, 2026 and cznic committed Apr 06, 2026

The non-recursive branch of mutexTry wrote mutex.id
with a plain store while mutexHeld/mutexNotheld
read it with atomic.LoadInt32.

Mixing atomic and non-atomic accesses to the
same word is a data race under Go's memory model.

Use atomic.StoreInt32, consistent with every other
write site (mutexEnter, mutexLeave, and the
recursive branch of mutexTry).

I have a regression test for this, but I'm not
convinced it is worth adding. It's a lot of code
and compute for an obvious fix that is unlikely to regress.

fix data race on mutex.id in mutexTry non-recursive path

See merge request !104

don't leak Backup.Commit dest conn on error

See merge request !105

Josh Bleecher Snyder authored Apr 04, 2026 and cznic committed Apr 06, 2026

The exec path treated SQLITE_ROW identically to SQLITE_DONE,
calling newResult after a single step.

For DML with RETURNING clauses that affect multiple rows,
this meant the statement was never stepped to completion,
causing sqlite3_changes() to return stale counts.

Continue stepping until SQLITE_DONE before reading the result.

This could manifest as a performance regression: we're doing strictly
more work by draining all the rows instead of abandoning them.
But as Russ Cox says, I can make anything fast if it doesn't need
to be correct.

This brings this Exec implementation in line with C sqlite3_exec
in several ways:

- Row draining: C sqlite3_exec loops sqlite3_step until rc != SQLITE_ROW,
  then finalizes.

- Result after completion: C sqlite3_exec  does finalize-after-loop.

- Unconditional drain: C sqlite3_exec with a NULL callback still
  steps through every row.

- Partial commit on interruption: C returns SQLITE_ABORT if the
  callback returns non-zero mid-drain, but already-stepped rows stay
  committed. (Use an explicit transaction for atomicity.)

- Multi-statement iteration: C's outer loop prepares and fully steps
  each sub-statement.

step Exec to completion for SQLITE_ROW

See merge request cznic/sqlite!106

See [GitLab issue #249](#249), thanks Emrecan BATI!

registerSingleModule set FiVersion = 1, but also populated
FxSavepoint, FxRelease, and FxRollbackTo. SQLite gates dispatch
of those callbacks on iVersion >= 2, so they were silently ignored.
Begin/Sync/Commit/Rollback (version 1 methods) worked fine,
which made the gap easy to miss.

iVersion was 1 from the initial commit.
Version 2 does not change any version-1 callback behavior.
The only behavioral change is that SQLite now dispatches the three
savepoint callbacks, and it still checks each function pointer is
non-zero before calling. The trampolines already handle modules that
don't implement the methods (type-assert, return SQLITE_OK).
So the bump to version 2 is safe.

Version 3 adds xShadowName.
Version 4 adds xIntegrity.
Neither callback is set, so those remain inert;
there's no need to upgrade past 2.

Extend the existing updater vtab test to implement Transactional
and verify that savepoint/release/rollbackto are dispatched.

set sqlite3_module.iVersion to 2 so savepoint callbacks fire

See merge request !107

Josh Bleecher Snyder authored Apr 06, 2026 and cznic committed Apr 07, 2026

The current code assumes a well-behavior io.Reader,
which works until it doesn't.

Add protection: convert f.Read(b) to io.ReadFull(f, b).
While we're here, use the new clear built-in.

Add test coverage, including tests that would have caught
the initial bugs and tests ensuring tail clearing.

fix vfsRead reader handling

See merge request !108

Josh Bleecher Snyder authored Apr 05, 2026 and cznic committed Apr 07, 2026

Two upcoming commits need a bit more control over
exactly which db is used to generate an error.

In anticipation of that, do some light refactoring.

While we're here handle db == 0 gracefully:
skip sqlite3_errmsg, which would misleadingly
return "out of memory" for a NULL handle.

extract errstrForDB from conn.errstr

See merge request !109

Per SQLite docs, sqlite3_backup_init stores errors
on the destination database handle.

In backup mode the destination is remoteConn.db; in
restore mode it is c.db. Previously errcode was always
read from remoteConn.db (wrong for restore) and errmsg
was always read from c.db via c.errstr (wrong for backup).
Read both from the actual destination handle.

read error from correct db handle on backup init failure

See merge request !111

Josh Bleecher Snyder authored Apr 05, 2026 and cznic committed Apr 08, 2026

When sqlite3_open_v2 fails, the SQLite docs require the caller to close
the handle it may have allocated. The previous code did not, leaking the
sqlite3 struct and potentially VFS-level state.

Additionally, newConn did not close libc.TLS on the openV2 failure path,
leaking C-heap memory.

Lastly, the error message was misleading: errstr called sqlite3_errmsg
with c.db == 0 (not yet set), which always returns "out of memory",
producing errors like "unable to open database file: out of memory (14)"
regardless of the actual cause.

Fix all of those.

fix openV2 handle leak, TLS leak, and misleading error on failed open

See merge request !112

 - Added `-DSQLITE_ENABLE_DBPAGE_VTAB` to the transpilation. See ["The
   SQLITE_DBPAGE Virtual Table"](https://www.sqlite.org/dbpage.html) for
   details.

Closes #250

Useful for on-the-fly introspection into queries.

Fixes #251

add ColumnInfo, for inspecting query columns

Closes #251

See merge request !113

Documents the transpilation-based architecture (lib/, vec/, vfs/), the
fragile modernc.org/libc version coupling, Makefile targets, debug build
tags, the GitLab-canonical / GitHub-mirror workflow, and the singleton
Driver registration model.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Ian Chechin authored May 13, 2026 and cznic committed May 21, 2026

Closes #219. Mirrors the existing FileControlPersistWAL pattern: takes
the schema name, allocates a 4-byte slot via the TLS allocator,
invokes Xsqlite3_file_control with the SQLITE_FCNTL_DATA_VERSION
opcode, and returns the resulting uint32.

The returned data version changes whenever the contents of the
database file change, so the typical caller polls it to invalidate
application-level caches.

Adds a regression test that opens a fresh file-backed database,
queries the version, performs a commit on the same connection,
performs another commit from a separate connection, and asserts the
version moves both times.

add FileControl.FileControlDataVersion wrapper for SQLITE_FCNTL_DATA_VERSION (#219)

Closes #219

See merge request !115

Ian Chechin authored May 21, 2026 and cznic committed May 21, 2026

Following the invitation in !115 review thread.

add Ian Chechin to AUTHORS and CONTRIBUTORS

See merge request !118

Ian Chechin authored May 12, 2026 and cznic committed May 22, 2026

The []driver.Value slice passed to user-defined-function scalar and
aggregate callbacks, and to vtab Filter / Insert / Update, was
allocated fresh on every invocation. For queries that fan out a UDF
over many rows this is the dominant per-row allocation, identified
as a hot spot in #226 with profile data showing ~13.5% of allocations
attributed to functionArgs.

The driver's contract on FunctionImpl.Scalar and
AggregateFunction.Step / WindowInverse already states the argument
values are not valid past the return of the user function, so the
slice itself can be reused safely.

Add a sync.Pool of *[]driver.Value and route the five existing call
sites through acquireUDFArgs / releaseUDFArgs:
  - funcTrampoline       (scalar UDFs)
  - stepTrampoline       (aggregate Step)
  - inverseTrampoline    (aggregate WindowInverse)
  - vtabFilterTrampoline (vtab Filter)
  - vtabUpdateTrampoline (vtab Insert / Update)

releaseUDFArgs zeroes each entry before returning the slice so any
heap references held in the previous invocation can be reclaimed.

Benchmark on 1000-row query with a 3-arg noop scalar UDF (Apple M3,
Go 1.25, -benchtime 3s -count 3):

  name                old time/op     new time/op     delta
  UDFArgsAllocation-8 213000 ns/op    205000 ns/op    -4%

  name                old alloc/op    new alloc/op    delta
  UDFArgsAllocation-8 118331 B/op     70376 B/op      -40%

  name                old allocs/op   new allocs/op   delta
  UDFArgsAllocation-8 6754 allocs/op  5754 allocs/op  -15%

The 1000 saved allocations per iteration match the expected savings:
one slice header per UDF invocation, times 1000 invocations per query.

Updates #226.

Ian Chechin authored May 21, 2026 and cznic committed May 22, 2026

After !114 pools the []driver.Value slice across UDF and vtab callbacks,
vtab Cursor.Filter and Updater.Insert/Update share the same "not valid
past return" contract as FunctionImpl.Scalar / AggregateFunction.Step /
WindowInverse. Document it explicitly so vtab implementations don't
silently retain references and corrupt later invocations.

Per cznic review on !114.

pool the []driver.Value slice in UDF and vtab callbacks (#226)

See merge request !114

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Ian Chechin authored May 13, 2026 and cznic committed May 26, 2026

The fix for #198 made (*conn).usable() return false whenever
sqlite3_is_interrupted reports the connection is interrupted, so
database/sql discards the connection after a context-cancelled query.
For file-backed databases that is fine, the data is on disk and a new
connection re-opens the same database. For in-memory databases the
connection IS the database, so dropping it loses the entire store -
re-introducing the regression originally fixed by !74 (issue #196).

Detect at open time whether the database is in-memory by checking the
output of sqlite3_db_filename and cache the result on the conn.
usable() short-circuits to true for those connections so the
post-interrupt one stays in the pool. File-backed connections keep
the existing behaviour - they are still reported unusable after an
interrupt.

Adds TestInMemoryDBSurvivesContextCancel with two subtests: one that
exercises the regression (insert rows, run a pre-cancelled query,
re-query and assert the rows are still there), and one that asserts
the file-backed path is still discarded after an explicit
sqlite3_interrupt.

Closes #196.

Ian Chechin authored May 25, 2026 and cznic committed May 26, 2026

Per cznic's review on !116, the previous in-memory subtest used
ExecContext with a pre-cancelled context, which short-circuits in
stmt.exec before Xsqlite3_interrupt is ever called, so the table-still-
present check passed even without the fix.

Rewrite the subtest to obtain a *conn via db.Conn().Raw(), call
sqlite3.Xsqlite3_interrupt directly, and assert c.usable() returns true
(matches the file-backed subtest's shape). Retain the end-to-end
QueryRow on the shared in-memory cache as a regression sanity check.
Verified locally that the subtest fails when the c.inMemory short-
circuit in (*conn).usable() is removed.

keep in-memory connections valid after a context-cancelled query (#196)

Closes #196

See merge request !116

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Ian Chechin authored May 25, 2026 and cznic committed May 26, 2026

Follow-up to !114 (#226). !114 pooled the []driver.Value slice header but
explicitly left the per-row TEXT/BLOB body copies in place because a
default-on zero-copy path would silently corrupt user code that retains
an argument across rows -- undetectable by -race (UDF execution is
sequential on one goroutine).

This commit adds VolatileArgs bool to FunctionImpl as a strict opt-in.
When true:
  - TEXT arguments are unsafe.String views into the SQLite-owned
    sqlite3_value_text buffer
  - BLOB arguments are unsafe.Slice views into the sqlite3_value_blob
    buffer
When false (the default for all existing call sites), behavior is
byte-for-byte identical to current master.

Plumbing: the flag is captured at registration into small wrapper structs
keyed in xFuncs.m / xAggregateFactories.m / xAggregateContext.m, so the
hot path is one extra field read rather than a second map lookup. The
five trampolines (funcTrampoline, stepTrampoline, inverse...

Per @cznic on !120:

1. The volatile branch of functionArgs returned []byte(nil) for empty
   BLOB args, while the non-volatile branch returned make([]byte, 0).
   A user comparing args[i] == nil would see different results depending
   on the flag, which is orthogonal to the volatility contract. Switch
   the volatile branch to make([]byte, 0) so the empty-BLOB shape is
   identical across both modes.

2. Document the within-callback re-entrancy hazard in the VolatileArgs
   docstring: a nested Query/Exec on the same connection during a
   volatile-args callback can cause SQLite to reuse the underlying
   value buffers, so a volatile string/[]byte read before the nested
   call may alias different bytes after it returns. Rare in practice,
   but useful to spell out alongside the cross-row retention rule.

TestVolatileArgsScalar and TestVolatileArgsAggregate stay green.

add FunctionImpl.VolatileArgs opt-in for zero-copy TEXT/BLOB args (#226)

See merge request !120

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Follow-up to !120 (#226). !120 added the FunctionImpl.VolatileArgs
opt-in for zero-copy TEXT/BLOB access on scalar and aggregate UDF
callbacks but left vtab Filter/Update on the non-volatile path,
flagged in the MR description as a candidate for a follow-up "if
there's demand".

This commit extends the same opt-in to:
  - Cursor.Filter (xFilter)
  - Updater.Insert / Updater.Update (xUpdate)

The contract on those callbacks already says "the vals/cols slice and
its entries are not valid past the return of this method;
implementations must copy any value they wish to retain", so the API
surface is unchanged for users who do not opt in; only the
body-allocation strategy differs when opt-in is set.

Opt-in is exposed as a new optional interface in package vtab:

    type VolatileArgsOpter interface {
        VolatileArgs() bool
    }

A Module that implements it and returns true gets zero-copy TEXT/BLOB
views for every Filter, Insert, and Upd...

extend VolatileArgs opt-in to vtab Filter and Updater Insert/Update

See merge request !121

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>