Commit cc920f9b authored by cznic's avatar cznic
Browse files

lib, vec: re-vendor, bump libc to v1.74.4, sweep the docs

Re-vendor the transpiled sources from ../libsqlite3 and ../libsqlite_vec.
SQLite stays at 3.53.3; what changes is that the amalgamation now carries
libsqlite3's sqlite_superjournal.patch, fixing an upstream 3.53.3
data-corruption bug in journal rollback. After a crash during the commit of a
multi-database (ATTACH) transaction the super-journal name and its checksum
can be left zeroed while the name length and the trailing magic survive; the
checksum is a plain byte sum, so an all-zero name still validates,
readSuperJournal() hands back a non-NULL pointer to an empty string, and
pager_playback() deletes the hot journal without replaying it. All 19 targets
carry the patch; master had it on none.

Verified by expanding both the old and the new trees and diffing per target:
17 of 19 targets differ by exactly that one line. linux/s390x additionally
picks up modernc.org/cc/v4 v4.29.1's MSB-first big-endian bit-field
allocation, it being this module's only big-endian target. linux/riscv64 was
regenerated on GCC 11.4.0 rather than 13.3.0, which only moves the COMPILER=
entry PRAGMA compile_options reports and drops some unexported predefined
macro constants; no SQLite code generation differs.

go.mod goes to the current releases throughout. modernc.org/libc lands on
v1.74.4 rather than the v1.74.3 ../libsqlite3 pins, that version being
retracted upstream for a freeaddrinfo lock leak that deadlocks name
resolution.

Documentation:

  - openbsd/amd64 and openbsd/arm64 join the supported platforms table. Both
    have been in builder.json's test matrix since January and are cross-built
    by make build_all_targets, but were never listed.
  - Document the vfs DSN query parameter on Driver.Open.
  - Rewrite "Debug and development versions". It described a GO_GENERATE
    environment variable that is no longer read anywhere and ccgo/v3; drop
    with it the //go:generate naming generator.go, deleted from this repo back
    at SQLite 3.45.1, which made go generate ./... fail.
  - Add the package doc comments vec and vfs were missing.
  - Correct two stale claims in CLAUDE.md.

make build_all_targets and make test are green (249 pass, 0 fail, 2 skip), as
are ./vfs/..., ./pcache/... and ./vtab/....

Co-Authored-By: default avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
parent 581eb450
Loading
Loading
Loading
Loading
+5 −1
Original line number Diff line number Diff line
# Changelog

 - 2026-07-30 v1.56.0:
 - 2026-08-03 v1.56.0:
     - Re-vendor the transpiled SQLite sources, picking up `modernc.org/libsqlite3`'s fix for an upstream **data-corruption bug in SQLite 3.53.3's journal rollback**. The SQLite version is unchanged at [3.53.3](https://sqlite.org/releaselog/3_53_3.html); what changes is that the amalgamation is now patched before it is transpiled. 3.53.3 reworked `readSuperJournal()` to return the super-journal name through a `char**` out-parameter, and `pager_playback()` now tests that pointer where it used to test `zSuper[0]`. A crash during the commit of a multi-database (ATTACH) transaction can leave the super-journal name and its checksum zeroed while the name length and the trailing magic survive; the checksum is a plain byte sum, so an all-zero name still validates and `readSuperJournal()` hands back a non-NULL pointer to an empty string. `pager_playback()` then calls `sqlite3OsAccess(pVfs, "", SQLITE_ACCESS_EXISTS)`, gets ENOENT, and deletes the hot journal without playing it back — leaving the database corrupted. This is not a transpilation artifact: a plain gcc build of the stock 3.53.3 amalgamation fails on the same bytes while 3.53.2 recovers them, and it is what has been making upstream's own `test/crash.test` fail intermittently, in roughly 2% of runs, on every platform. The patch restores the pre-3.53.3 behaviour of reporting a `(nul)` super-journal name and will be dropped once upstream ships its own fix. Every supported target carries it.
     - Two targets change beyond that patch. On `linux/s390x` the regenerated transpile allocates C bit-fields MSB-first, as the big-endian platform ABI requires, rather than LSB-first; this comes from `modernc.org/cc/v4` v4.29.1 and touches bit-field accesses throughout the SQLite core, s390x being this module's only big-endian target. On `linux/riscv64` the transpile was regenerated on a host running GCC 11.4.0 where the previous one used GCC 13.3.0, which drops a handful of unexported compiler-predefined macro constants (the `__FLT16_*` family, `__DBL_IS_IEC_60559__` and friends) and changes the `COMPILER=gcc-13.3.0` entry `PRAGMA compile_options` reports to `COMPILER=gcc-11.4.0`; no SQLite code generation differs. Every other target's generated code is byte-identical to v1.55.0 apart from the journal-rollback patch above.
     - Bump the pinned `modernc.org/libc` to v1.74.4, and the remaining dependencies to their current releases. v1.74.2 and v1.74.3 are retracted upstream — a `freeaddrinfo` lock leak that deadlocks name resolution — and v1.74.4 is the fix. As always, downstream modules must pin the exact `modernc.org/libc` version this module's `go.mod` pins (see [GitLab issue #177](https://gitlab.com/cznic/sqlite/-/issues/177)).
     - Documentation sweep. `openbsd/amd64` and `openbsd/arm64` join the supported platforms table in the package documentation: both have been in the builder test matrix since January and are cross-built by `make build_all_targets`, but had never been listed. The `vfs` DSN query parameter — which names a VFS registered with SQLite, such as one returned by `vfs.New` — is now documented alongside the other DSN parameters on `Driver.Open`. The "Debug and development versions" section no longer describes a `GO_GENERATE` environment variable and a `go generate` that this repository has not had since `generator.go` moved to `modernc.org/libsqlite3`; it now points at that repository and `make vendor` instead, and the stale `//go:generate` directive naming the removed file is dropped with it. `modernc.org/sqlite/vec` and `modernc.org/sqlite/vfs` gained the package doc comments they were missing, so both finally carry a synopsis on pkg.go.dev. Documentation only; no behavior changes.
     - Add `NewConnector`, returning a `database/sql/driver.Connector` for use with `sql.OpenDB`. It opens the same connections `sql.Open("sqlite", dsn)` does, from the same registered driver, so every function, collation, connection hook and virtual table module registered through this package applies to them. It exists for callers that need to interpose on the physical connections `database/sql` opens — tracing, metrics, connection-scoped setup — which `sql.Open` gives no access to: such a caller can embed the returned `Connector`, override `Connect`, and pass its own wrapper to `sql.OpenDB`. Previously the only way to reach the registered driver was the `db, _ := sql.Open("sqlite", ""); drv := db.Driver(); db.Close()` idiom, which works only because `sql.Open` does not connect and this driver does not implement `driver.DriverContext`; and the only way to get a wrapper into a `*sql.DB` was `sql.Register`, which is process-global, panics on a name it has already seen, and cannot be undone, so a library had to invent a unique driver name per configuration. `sql.OpenDB` registers nothing. Constructing a `&sqlite.Driver{}` is not an alternative — its fields are unexported, so it carries none of the registrations. `NewConnector` checks the DSN only as far as it can without opening a database — a query string that does not parse, and conflicting `vfs` parameters; everything else continues to be validated when the connection is opened, so an unknown parameter or an out-of-range value is reported by `Connect` rather than at construction. Nothing about the existing `sql.Open` path changes: `*Driver` deliberately still does not implement `driver.DriverContext`, so `sql.Open` remains lazy and DSN errors continue to surface where they always have. A runnable sample is in `examples/connector`. Resolves [GitLab issue #253](https://gitlab.com/cznic/sqlite/-/issues/253), thanks Alessandro Segala (@ItalyPaleAle)!
     - Document that a caller-constructed `sqlite.Driver` is not the driver this package registers as `"sqlite"`. Its fields are unexported, so it starts with no functions, collations or connection hooks and the only way to give it any is its own `RegisterConnectionHook` method; the package-level `Register*` functions always apply to the registered driver. Connections such a `Driver` opens therefore run without the package-level functions and collations — and because a registered function silently replaces a SQLite built-in of the same name, a `Driver` you construct can evaluate `upper(x)`, `date(x)` and the like differently from one opened through `sql.Open`. Virtual table modules are the one exception: they are held process-globally and reach every `Driver`. Constructing one remains supported for the private-hook pattern — a driver registered under a name of its own with `sql.Register` so its connection hooks apply only to its own connections — and is otherwise best avoided in favour of `sql.Open` or `NewConnector`. Documentation only; no behavior changes.

+3 −2
Original line number Diff line number Diff line
@@ -13,7 +13,7 @@ The hand-written Go on top of that transpiled core implements the `database/sql/
- `sqlite.go`, `conn.go`, `driver.go`, `stmt.go`, `rows.go`, `tx.go`, `backup.go`, `error.go`, `result.go`, `convert.go` — hand-written `database/sql/driver` implementation calling into `lib/`.
- `vtab.go`, `pre_update_hook.go`, `fcntl.go`, `mutex.go` — Go-facing extensions wired to SQLite hooks/trampolines.
- `lib/` — transpiled SQLite 3.53.3. One `sqlite_<goos>_<goarch>.go` per supported triple; `defs.go`, `hooks.go`, `hooks_linux_arm64.go`, `mutex.go`, plus `libsqlite3_freebsd.go`/`libsqlite3_windows.go` hold hand-written patches that augment the generated code. Import as `sqlite3 "modernc.org/sqlite/lib"`.
- `vec/` — transpiled `sqlite-vec` v0.1.9, auto-registers via `sqlite3_auto_extension` in `patches.go` on package init. Activate by blank-importing: `_ "modernc.org/sqlite/vec"`. Not all platforms have a `vec_*.go` (e.g. no `linux/s390x` in `vec_test.go`'s `//go:build`).
- `vec/` — transpiled `sqlite-vec` v0.1.9, auto-registers via `sqlite3_auto_extension` in `patches.go` on package init. Activate by blank-importing: `_ "modernc.org/sqlite/vec"`. Covers the same 19 targets `lib/` does; `vec_test.go`'s `//go:build` constrains by GOOS only.
- `vfs/` — exposes a Go `fs.FS` as a read-only SQLite VFS. `vfs.New(fsys)` returns a registered VFS name; open with `?vfs=<name>`. C side is transpiled per platform from `vfs/c/vfs.c` via the `vfs/Makefile`.
- `vtab/` — Go-facing virtual-table API (no dependency on the transpiled C). `vtab.RegisterModule(db, name, module)` registers modules on **new connections only**; the bridge to C lives in the top-level `vtab.go`. See `vtab/doc.go` for the contract (Updater/Renamer/Transactional optional interfaces, re-entrancy rules, ArgIndex/Omit semantics).
- `vendor_libs/main.go` (build tag `none`) — regeneration tool. Reads transpiled `ccgo_<goos>_<goarch>.go` from sibling repos `../libsqlite3` and `../libsqlite_vec`, rewrites package names and imports, and writes `lib/sqlite_*.go` / `vec/vec_*.go`. Invoked by `make vendor`.
@@ -37,7 +37,8 @@ Single test: `go test -v -run TestScalar` (pattern is a regexp; tests live in `a
Build/debug tags:
- `-tags=sqlite.dmesg` — enables this package's `dmesg(...)` (writes to `/tmp/libc.log`); see `dmesg.go` / `nodmesg.go`.
- `-tags=libc.dmesg` — enables debug logs from `modernc.org/libc` (must be combined with patching `libc` itself — see the worked example in `doc.go`).
- `GO_GENERATE=-DSQLITE_DEBUG,-DSQLITE_MEM_DEBUG` for `go generate` to produce a debug-instrumented transpilation (requires `modernc.org/ccgo/v4` installed locally).

There is no `go generate` in this repo — `generator.go` lives in `../libsqlite3`, which owns the transpilation and its SQLite compile-time options. To produce a debug-instrumented transpilation, change the options there, `make generate` in that repo, then `make vendor` here.

## Fragile `modernc.org/libc` coupling

+15 −11
Original line number Diff line number Diff line
@@ -50,6 +50,8 @@
//	linux	ppc64le 3.53.3
//	linux	riscv64 3.53.3
//	linux	s390x   3.53.3
//	openbsd	amd64   3.53.3
//	openbsd	arm64   3.53.3
//	windows	386     3.53.3
//	windows	amd64   3.53.3
//	windows	arm64   3.53.3
@@ -89,18 +91,18 @@
//
// # Debug and development versions
//
// A comma separated list of options can be passed to `go generate` via the
// environment variable GO_GENERATE. Some useful options include for example:
// The transpiled SQLite sources under lib/, and the sqlite-vec sources under
// vec/, are not generated in this repository. They are produced by
// modernc.org/libsqlite3 and modernc.org/libsqlite_vec respectively, which own
// the transpilation and the SQLite compile-time options it uses, and are
// copied here by
//
//	-DSQLITE_DEBUG
//	-DSQLITE_MEM_DEBUG
//	-ccgo-verify-structs
//	$ make vendor
//
// To create a debug/development version, issue for example:
//
//	$ GO_GENERATE=-DSQLITE_DEBUG,-DSQLITE_MEM_DEBUG go generate
//
// Note: To run `go generate` you need to have modernc.org/ccgo/v3 installed.
// which reads them from checkouts of those two repositories placed next to
// this one. To build a debug or otherwise modified version, adjust the
// compile-time options in modernc.org/libsqlite3, regenerate there with 'make
// generate', and vendor the result here.
//
// # Hacking
//
@@ -180,7 +182,9 @@
//	 }
//	0:jnml@e5-1650:~/src/modernc.org/libc$
//
// We need to tell the Go build system to use our local, patched/debug libc:
// We need to tell the Go build system to use our local, patched/debug libc.
// 'make work' sets up a go.work covering this and the sibling repositories;
// by hand it is:
//
//	0:jnml@e5-1650:~/src/modernc.org/sqlite$ go work use $(go env GOPATH)/src/modernc.org/libc
//	0:jnml@e5-1650:~/src/modernc.org/sqlite$ go work use .
+8 −0
Original line number Diff line number Diff line
@@ -178,6 +178,14 @@ func newDriver() *Driver { return d }
// driver's *Error is unchanged in either mode. The parameter is parsed
// before sqlite3_open_v2 so open-time errors are covered. See
// https://gitlab.com/cznic/sqlite/-/issues/230.
//
// vfs: The name of the SQLite VFS to open the database with. Note the absent
// underscore prefix: this is the same parameter SQLite recognizes in a file:
// URI, and its value is passed on as the sqlite3_open_v2 zVfs argument. It
// selects any VFS registered with SQLite, in particular one returned by
// [modernc.org/sqlite/vfs.New], which exposes a Go fs.FS as a read-only VFS.
// When absent or empty the default VFS is used. Supplying the parameter more
// than once with values that differ is an error.
func (d *Driver) Open(name string) (conn driver.Conn, err error) {
	if dmesgs {
		defer func() {
+4 −4
Original line number Diff line number Diff line
@@ -3,17 +3,17 @@ module modernc.org/sqlite
go 1.25.0

require (
	github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e
	golang.org/x/sys v0.46.0
	github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3
	golang.org/x/sys v0.47.0
	modernc.org/fileutil v1.4.0
	modernc.org/libc v1.74.1
	modernc.org/libc v1.74.4
	modernc.org/mathutil v1.7.1
)

require (
	github.com/dustin/go-humanize v1.0.1 // indirect
	github.com/google/uuid v1.6.0 // indirect
	github.com/mattn/go-isatty v0.0.20 // indirect
	github.com/mattn/go-isatty v0.0.24 // indirect
	github.com/ncruces/go-strftime v1.0.0 // indirect
	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
	modernc.org/memory v1.11.0 // indirect
Loading