Commit 47d0960a authored by cznic's avatar cznic
Browse files

Merge branch 'driver-registration' into 'master'

sqlite: let a caller-constructed Driver register its own functions, collations and modules

See merge request cznic/sqlite!135
parents 15039fd3 9ed2aad5
Loading
Loading
Loading
Loading
+3 −1
Original line number Diff line number Diff line
# Changelog

 - 2026-08-09 v1.57.0:
 - 2026-08-15 v1.57.0:
     - Add an opt-in `_defensive` DSN query parameter that turns on SQLite's defensive mode for the connection, disabling the SQL-level features that let ordinary statements deliberately corrupt the database file. When `_defensive=1` (or any `strconv.ParseBool` true value) is supplied, the driver calls `sqlite3_db_config` with `SQLITE_DBCONFIG_DEFENSIVE` immediately after `sqlite3_open_v2` and before every other parameter is applied, so the PRAGMAs the driver itself runs, the `_pragma` list, and every statement the caller prepares are all subject to it. On such a connection `PRAGMA writable_schema=ON`, `PRAGMA journal_mode=OFF` and `PRAGMA schema_version=N` become silent no-ops, and writes to a virtual table's shadow tables (fts5's `_data`, `_idx` and so on) and to `sqlite_dbpage` fail with "table ... may not be modified"; reading those tables, ordinary use of the virtual tables that own them, and `VACUUM` are unaffected. The flag has no PRAGMA equivalent, so `sqlite3_db_config` — and therefore a DSN parameter — is the only way to reach it short of dropping to `modernc.org/sqlite/lib`. The value is parsed before `sqlite3_open_v2`, so an invalid one fails the connection without creating the database file, and the parameter must appear at most once: a repeated `_defensive` is an error rather than letting the first value silently win. Absence of the parameter, or `_defensive=0`, leaves SQLite's default behavior unchanged; existing DSNs continue to work byte-for-byte. Two limits are worth stating plainly, since the name invites more confidence than the flag earns. Defensive mode is a hardening measure, not a sandbox for hostile database files: it is one of several steps [SQLite recommends](https://www.sqlite.org/security.html) for that purpose, and this build compiles with neither `SQLITE_TRUSTED_SCHEMA=0` nor `SQLITE_DQS=0` and exposes no authorizer. And it is a property of the connection, not of the database file — a second handle opened on the same file without the parameter is unrestricted.
     - Reject the one DSN combination defensive mode would otherwise swallow in silence. `_defensive=1` together with `_journal_mode=OFF` (or `_journal=OFF`) now fails the connection instead of opening one in which neither parameter was honoured: SQLite turns `PRAGMA journal_mode=OFF` into a no-op that still reports success, so the driver would have accepted the mode, executed it, and left the journal untouched without telling anyone. The check runs in the validation phase introduced in v1.55.0, before any statement executes, so a rejected DSN cannot leave the database half-configured. `_pragma` remains the exception it has always been: `_pragma=journal_mode(OFF)` alongside `_defensive=1` still runs and is still silently ignored by SQLite. Only DSNs using `_defensive` can be affected, and that parameter is new, so no DSN that opened before changes behavior.
     - See [GitHub pull request #6](https://github.com/modernc-org/sqlite/pull/6), thanks wsman!
     - Ship the sqlite-vec license notice this module has been missing. `modernc.org/sqlite/vec` has bundled the transpiled [sqlite-vec](https://github.com/asg017/sqlite-vec) sources since v1.47.0, but the module carried only its own BSD-3-Clause `LICENSE` and the public-domain SQLite notice. sqlite-vec is Copyright (c) 2024 Alex Garcia, dual-licensed Apache-2.0 OR MIT and used here under MIT, whose terms require the copyright and permission notice to accompany substantial portions of the software — which 2.8 MB of transpiled `vec/` plainly is. The notice now ships as `LICENSE-SQLITE_VEC` in the module root, byte-identical to the `LICENSE-MIT` in the upstream v0.1.9 archive and named after the file `modernc.org/libsqlite_vec` extracts it into. Attribution was never absent — `vec`'s package documentation has named the extension, pinned the version and linked upstream — but the license text itself was, and the omission was ours: `vendor_libs/main.go` copied the per-target transpiles and nothing else. It now copies the notice alongside them and fails the vendoring run if it cannot, so a `make vendor` can no longer quietly drop it. The `vec` package documentation gained a License section recording that the package is under a different license from the rest of this module.
     - **The SQLite notice is renamed from `SQLITE-LICENSE` to `LICENSE-SQLITE`**; update any direct links to it. Its contents are unchanged and SQLite remains public domain. The name now matches both the new `LICENSE-SQLITE_VEC` beside it and the `LICENSE-<upstream>` convention every other modernc.org repository follows, but it is more than cosmetic: `go mod vendor` selects the files it copies into a downstream `vendor/` tree by matching each name against a fixed list of prefixes — `LICENSE` among them — so a name merely *ending* in `LICENSE` was never propagated. Both bundled notices now travel with the code into vendored builds, which is where the MIT terms on `vec/` keep applying. No code changes; no behavior changes.
     - Let a caller-constructed `Driver` register its own functions, collations and virtual table modules. `Driver` has always held four categories of registration state, but only `RegisterConnectionHook` could put anything on a constructed one: functions and collations were reachable through the package-level API alone, and modules through the package-level driver only, which left the `modules` field written and read through that instance and so process-global state wearing a per-instance field. `Driver` now has `RegisterFunction`, `RegisterScalarFunction`, `RegisterDeterministicScalarFunction`, `RegisterCollationUtf8` and `RegisterModule`, plus `Must*` variants of the first four, each registering on that `Driver` alone; the methods are safe to call concurrently, and the zero `Driver` is ready to use as-is. `vtab.RegisterModule` also honours its `db` argument now: a non-nil `db` registers on the driver backing it when that driver implements the new `vtab.ModuleRegisterer`, while a nil `db` keeps targeting the driver this package registers as `sqlite`. One existing pattern changes behavior, narrowly and loudly: `vtab.RegisterModule(db, ...)` where `db` was opened on a caller-constructed `Driver` used to discard the `db` argument and land on the `sqlite` driver, reaching every connection in the process; it now lands on the constructed driver alone, so a `sql.Open("sqlite")` connection that used to resolve such a module gets `no such module` instead. The same pattern is also the one way an existing program could hold one module name on both a constructed `Driver` and the package-level one: there the first of the two registrations used to win and the second was refused as already registered, whereas now the package-level implementation wins on the constructed `Driver`'s connections regardless of the order they ran in. Reaching that case at all means the program ignored an error the older version returned. Two smaller deviations round out the list: `Driver.RegisterModule` reports no error for such a collision, and `vtab.RegisterModule` now validates its name and module arguments before the not-implemented check, so a call with an empty name that returned `vtab: RegisterModule not wired into engine` outside this driver returns `vtab: module name must be non-empty` instead. Everything else is additive against v1.56.0: the package-level registration functions target the same driver they always did, connections still receive every module registered through the package-level path whichever `Driver` opened them, and a `db` opened on the `sqlite` driver resolves to that same driver. The isolating change discussed in [GitLab issue #254](https://gitlab.com/cznic/sqlite/-/issues/254) is deliberately not made here.
     - See [GitLab merge request #135](https://gitlab.com/cznic/sqlite/-/merge_requests/135), thanks Ian Chechin!

 - 2026-08-03 v1.56.0:
     - Re-vendor the transpiled SQLite sources, picking up `modernc.org/libsqlite3`'s fix for an upstream **data-corruption bug in SQLite 3.53.3's journal rollback**. The SQLite version is unchanged at [3.53.3](https://sqlite.org/releaselog/3_53_3.html); what changes is that the amalgamation is now patched before it is transpiled. 3.53.3 reworked `readSuperJournal()` to return the super-journal name through a `char**` out-parameter, and `pager_playback()` now tests that pointer where it used to test `zSuper[0]`. A crash during the commit of a multi-database (ATTACH) transaction can leave the super-journal name and its checksum zeroed while the name length and the trailing magic survive; the checksum is a plain byte sum, so an all-zero name still validates and `readSuperJournal()` hands back a non-NULL pointer to an empty string. `pager_playback()` then calls `sqlite3OsAccess(pVfs, "", SQLITE_ACCESS_EXISTS)`, gets ENOENT, and deletes the hot journal without playing it back — leaving the database corrupted. This is not a transpilation artifact: a plain gcc build of the stock 3.53.3 amalgamation fails on the same bytes while 3.53.2 recovers them, and it is what has been making upstream's own `test/crash.test` fail intermittently, in roughly 2% of runs, on every platform. The patch restores the pre-3.53.3 behaviour of reporting a `(nul)` super-journal name and will be dropped once upstream ships its own fix. Every supported target carries it.
+2 −2
Original line number Diff line number Diff line
@@ -15,7 +15,7 @@ The hand-written Go on top of that transpiled core implements the `database/sql/
- `lib/` — transpiled SQLite 3.53.3. One `sqlite_<goos>_<goarch>.go` per supported triple; `defs.go`, `hooks.go`, `hooks_linux_arm64.go`, `mutex.go`, plus `libsqlite3_freebsd.go`/`libsqlite3_windows.go` hold hand-written patches that augment the generated code. Import as `sqlite3 "modernc.org/sqlite/lib"`.
- `vec/` — transpiled `sqlite-vec` v0.1.9, auto-registers via `sqlite3_auto_extension` in `patches.go` on package init. Activate by blank-importing: `_ "modernc.org/sqlite/vec"`. Covers the same 19 targets `lib/` does; `vec_test.go`'s `//go:build` constrains by GOOS only.
- `vfs/` — exposes a Go `fs.FS` as a read-only SQLite VFS. `vfs.New(fsys)` returns a registered VFS name; open with `?vfs=<name>`. C side is transpiled per platform from `vfs/c/vfs.c` via the `vfs/Makefile`.
- `vtab/` — Go-facing virtual-table API (no dependency on the transpiled C). `vtab.RegisterModule(db, name, module)` registers modules on **new connections only**; the bridge to C lives in the top-level `vtab.go`. See `vtab/doc.go` for the contract (Updater/Renamer/Transactional optional interfaces, re-entrancy rules, ArgIndex/Omit semantics).
- `vtab/` — Go-facing virtual-table API (no dependency on the transpiled C). `vtab.RegisterModule(db, name, module)` registers modules on **new connections only**; a nil `db` targets the driver registered as `sqlite`, a non-nil `db` the driver backing it (via `vtab.ModuleRegisterer`). The bridge to C lives in the top-level `vtab.go`. See `vtab/doc.go` for the contract (Updater/Renamer/Transactional optional interfaces, re-entrancy rules, ArgIndex/Omit semantics).
- `vendor_libs/main.go` (build tag `none`) — regeneration tool. Reads transpiled `ccgo_<goos>_<goarch>.go` from sibling repos `../libsqlite3` and `../libsqlite_vec`, rewrites package names and imports, and writes `lib/sqlite_*.go` / `vec/vec_*.go`. Invoked by `make vendor`.
- `examples/` — runnable samples: `example1`, `connector`, `vtab_basic`, `vtab_csv`, `vtab_match`, `vtab_regexp`.
- `addport.go`, `issue198/`, `issue120.diff` — porting/regression scaffolding kept around for reference; not built.
@@ -54,6 +54,6 @@ When debugging into `libc`, use `make work` (or a manual `go work init && go wor

## Driver registration model

`init()` in `sqlite.go` calls `sql.Register("sqlite", newDriver())` with a single package-level `*Driver` (`var d` in `driver.go`). Global UDFs (`RegisterFunction`, `RegisterScalarFunction`, `RegisterDeterministicScalarFunction`), collations (`RegisterCollationUtf8`), connection hooks (`Driver.RegisterConnectionHook`), and vtab modules (`vtab.RegisterModule`) all attach to that singleton and are applied to every connection opened **afterwards**. Registrations made after a connection is open do not affect that connection — open a new one. This applies in particular to vtab modules; see `driver.go:120` and `vtab/doc.go`.
`init()` in `sqlite.go` calls `sql.Register("sqlite", defaultDriver())` with a single package-level `*Driver` (`var d` in `driver.go`). The package-level registration functions (`RegisterFunction`, `RegisterScalarFunction`, `RegisterDeterministicScalarFunction`, `RegisterCollationUtf8`, `RegisterConnectionHook`) attach to that driver alone; its vtab modules — `vtab.RegisterModule` with a nil `db` — are the one category held process-globally and reach every driver's connections. A caller-constructed `Driver` carries its own registrations through the mirroring `*Driver` methods (`RegisterFunction`, `RegisterScalarFunction`, `RegisterDeterministicScalarFunction`, `RegisterCollationUtf8`, `RegisterConnectionHook`, `RegisterModule`, plus `Must*` variants), and `vtab.RegisterModule` with a non-nil `db` lands on the driver backing that `db`. Same-name module on both: the package-level implementation wins on that driver's connections. All registration applies to connections opened **afterwards**; registrations made after a connection is open do not affect it — open a new one. See the `Driver` type doc in `driver.go` and `vtab/doc.go`.

DSN query params are parsed in `conn.go`/`driver.go`: `_pragma`, `_time_format`, `_time_integer_format`, `_inttotime`, `_texttotime`, `_timezone`, `_txlock`, plus `vfs=<name>` to select a VFS registered via `vfs.New`.
+1 −1
Original line number Diff line number Diff line
@@ -38,7 +38,7 @@ Virtual Tables (vtab)

The driver exposes a Go API to implement SQLite virtual table modules in pure Go via the `modernc.org/sqlite/vtab` package. This lets you back SQL tables with arbitrary data sources (e.g., vector indexes, CSV files, remote APIs) and integrate with SQLite’s planner.

- Register: `vtab.RegisterModule(db, name, module)`. Registration applies to new connections only.
- Register: `vtab.RegisterModule(db, name, module)`. A nil `db` registers on the driver this package registers as `sqlite`, whose modules reach every connection in the process; a non-nil `db` registers on the driver backing it, so a `db` opened on a caller-constructed `sqlite.Driver` keeps its modules to that driver's connections. Registration applies to new connections only.
- Schema declaration: Call `ctx.Declare("CREATE TABLE <name>(<cols...>)")` within `Create` or `Connect`. The driver does not auto-declare schemas, enabling dynamic schemas.
- Module arguments: `args []string` passed to `Create/Connect` are configuration parsed from `USING module(...)`. They are not treated as columns unless your module chooses to.
- Planning (BestIndex):
+147 −16
Original line number Diff line number Diff line
@@ -7,14 +7,18 @@ package sqlite // import "modernc.org/sqlite"
import (
	"database/sql/driver"
	"fmt"
	"sync"

	sqlite3 "modernc.org/sqlite/lib"
	"modernc.org/sqlite/vtab"
)

// Driver implements database/sql/driver.Driver.
//
// Registration functions and methods must be called before the first call to
// Open.
// Open. The methods are safe to call concurrently with each other, so a
// *Driver can be handed out for several packages to fill from their init
// functions; the ordering requirement against Open still stands.
//
// Most code has no use for this type. sql.Open("sqlite", dsn) and
// [NewConnector] both go through the driver this package registers as
@@ -23,21 +27,37 @@ import (
// [RegisterCollationUtf8], [RegisterConnectionHook] and
// [vtab.RegisterModule].
//
// A Driver a caller constructs is not equivalent to that one. Its fields are
// unexported, so it starts out with no functions, collations or connection
// hooks, and the only way to give it any is [Driver.RegisterConnectionHook];
// the package-level registration functions always apply to the registered
// driver, never to a constructed one. Connections it opens therefore run
// without the package-level functions and collations -- and where such a
// registration overrides a SQLite built-in of the same name, they run with
// SQLite's built-in in force instead. Virtual table modules are the one
// exception: they are held process-globally and reach every Driver.
// A Driver a caller constructs is not equivalent to that one. It starts out
// empty, and the package-level registration functions always apply to the
// registered driver, never to a constructed one. Connections it opens
// therefore run without the package-level functions and collations -- and
// where such a registration overrides a SQLite built-in of the same name, they
// run with SQLite's built-in in force instead. Virtual table modules are the
// one exception: those registered through the package-level path are held
// process-globally and reach every Driver.
//
// Constructing one is supported for the private-hook pattern: a driver
// registered under a name of its own with sql.Register, so that its connection
// hooks apply to its own connections rather than to every connection in the
// process. Prefer sql.Open or NewConnector for anything else.
// A constructed Driver is filled in through its own methods, each of which
// registers on that Driver alone: [Driver.RegisterFunction],
// [Driver.RegisterScalarFunction],
// [Driver.RegisterDeterministicScalarFunction],
// [Driver.RegisterCollationUtf8], [Driver.RegisterConnectionHook] and
// [Driver.RegisterModule]. The zero Driver is ready to use; there is no
// constructor. What it registers stays on it, so two constructed Drivers do
// not see each other's registrations and neither leaks into the package-level
// driver. Modules it registers itself are installed in addition to the
// process-global ones, not instead of them, and where both registered the
// same name the package-level implementation wins on its connections.
//
// Constructing one is supported for the private-registration pattern: a driver
// registered under a name of its own with sql.Register, so that its functions,
// collations, modules and connection hooks apply to its own connections rather
// than to every connection in the process. Prefer sql.Open or NewConnector for
// anything else.
type Driver struct {
	// mu guards the registration state below against concurrent
	// registrations. Open reads that state without it, which the ordering
	// requirement in the type documentation makes safe.
	mu sync.Mutex
	// user defined functions that are added to every new connection on Open
	udfs map[string]*userDefinedFunction
	// collations that are added to every new connection on Open
@@ -56,7 +76,7 @@ var d = &Driver{
	modules:         make(map[string]vtab.Module, 0),
}

func newDriver() *Driver { return d }
func defaultDriver() *Driver { return d }

// Open returns a new connection to the database. The name is a string in a
// driver-specific format.
@@ -253,7 +273,7 @@ func (d *Driver) Open(name string) (conn driver.Conn, err error) {
	// Note: vtab module registration applies to new connections only. If a
	// module is registered after a connection has been opened, that existing
	// connection will not see the module; open a new connection to use it.
	if err := c.registerModules(); err != nil {
	if err := c.registerModules(d); err != nil {
		c.Close()
		return nil, err
	}
@@ -268,5 +288,116 @@ func (d *Driver) Open(name string) (conn driver.Conn, err error) {
// sql.Open and [NewConnector] hand out, use the package-level
// [RegisterConnectionHook].
func (d *Driver) RegisterConnectionHook(fn ConnectionHookFn) {
	d.mu.Lock()
	defer d.mu.Unlock()

	d.connectionHooks = append(d.connectionHooks, fn)
}

// RegisterFunction is like the package-level [RegisterFunction] but registers
// the function on d alone, so it reaches only the connections d opens. See
// [Driver] for when to prefer this over the package-level form.
func (d *Driver) RegisterFunction(
	zFuncName string,
	impl *FunctionImpl,
) (err error) {
	if dmesgs {
		defer func() {
			dmesg("d %p, zFuncName %q, impl %p: err %v", d, zFuncName, impl, err)
		}()
	}
	return d.registerFunction(zFuncName, impl)
}

// MustRegisterFunction is like [Driver.RegisterFunction] but panics on error.
func (d *Driver) MustRegisterFunction(
	zFuncName string,
	impl *FunctionImpl,
) {
	if err := d.RegisterFunction(zFuncName, impl); err != nil {
		panic(err)
	}
}

// RegisterScalarFunction is like the package-level [RegisterScalarFunction]
// but registers the function on d alone, so it reaches only the connections d
// opens. See [Driver] for when to prefer this over the package-level form.
func (d *Driver) RegisterScalarFunction(
	zFuncName string,
	nArg int32,
	xFunc func(ctx *FunctionContext, args []driver.Value) (driver.Value, error),
) (err error) {
	if dmesgs {
		defer func() {
			dmesg("d %p, zFuncName %q, nArg %v, xFunc %p: err %v", d, zFuncName, nArg, xFunc, err)
		}()
	}
	return d.registerFunction(zFuncName, &FunctionImpl{NArgs: nArg, Scalar: xFunc, Deterministic: false})
}

// MustRegisterScalarFunction is like [Driver.RegisterScalarFunction] but
// panics on error.
func (d *Driver) MustRegisterScalarFunction(
	zFuncName string,
	nArg int32,
	xFunc func(ctx *FunctionContext, args []driver.Value) (driver.Value, error),
) {
	if err := d.RegisterScalarFunction(zFuncName, nArg, xFunc); err != nil {
		panic(err)
	}
}

// RegisterDeterministicScalarFunction is like the package-level
// [RegisterDeterministicScalarFunction] but registers the function on d alone,
// so it reaches only the connections d opens. See [Driver] for when to prefer
// this over the package-level form.
func (d *Driver) RegisterDeterministicScalarFunction(
	zFuncName string,
	nArg int32,
	xFunc func(ctx *FunctionContext, args []driver.Value) (driver.Value, error),
) (err error) {
	if dmesgs {
		defer func() {
			dmesg("d %p, zFuncName %q, nArg %v, xFunc %p: err %v", d, zFuncName, nArg, xFunc, err)
		}()
	}
	return d.registerFunction(zFuncName, &FunctionImpl{NArgs: nArg, Scalar: xFunc, Deterministic: true})
}

// MustRegisterDeterministicScalarFunction is like
// [Driver.RegisterDeterministicScalarFunction] but panics on error.
func (d *Driver) MustRegisterDeterministicScalarFunction(
	zFuncName string,
	nArg int32,
	xFunc func(ctx *FunctionContext, args []driver.Value) (driver.Value, error),
) {
	if err := d.RegisterDeterministicScalarFunction(zFuncName, nArg, xFunc); err != nil {
		panic(err)
	}
}

// RegisterCollationUtf8 is like the package-level [RegisterCollationUtf8] but
// registers the collation on d alone, so it reaches only the connections d
// opens. See [Driver] for when to prefer this over the package-level form.
func (d *Driver) RegisterCollationUtf8(
	zName string,
	impl func(left, right string) int,
) (err error) {
	if dmesgs {
		defer func() {
			dmesg("d %p, zName %q, impl %p: err %v", d, zName, impl, err)
		}()
	}
	return d.registerCollation(zName, impl, sqlite3.SQLITE_UTF8)
}

// MustRegisterCollationUtf8 is like [Driver.RegisterCollationUtf8] but panics
// on error.
func (d *Driver) MustRegisterCollationUtf8(
	zName string,
	impl func(left, right string) int,
) {
	if err := d.RegisterCollationUtf8(zName, impl); err != nil {
		panic(err)
	}
}
+343 −0

File added.

Preview size limit exceeded, changes collapsed.

Loading