Loading
Commits on Source 30
-
cznic authoredThe Driver.Open docstring listed only the keys the driver interprets itself, the underscore-prefixed ones plus vfs, and never said that a file: DSN also carries SQLite's own URI parameters (mode, cache, immutable, nolock, psow, modeof) because every connection is opened with SQLITE_OPEN_URI. The question has been asked in #48, #110, #162, #213 and now #257. The docstring now describes both DSN forms and what each does with the query: a plain file name has it stripped before SQLite sees the name, matching github.com/mattn/go-sqlite3, so "/path?mode=ro" silently opens read-write and creates a missing file, while "file:/path?mode=ro" is really read-only. It also explains what mode=ro does, how it differs from _query_only=1, and that a driver key which has to write the database, such as _journal_mode=WAL on a non-WAL database, fails the open of a mode=ro connection with SQLITE_READONLY. Every statement was checked against the current library with a probe program. doc.go points at Driver.Open from the connecting example, and CHANGELOG.md records the change under v1.59.1. Updates #257 Co-Authored-By:
Claude Fable 5.1 <noreply@anthropic.com>
-
cznic authored
Entries had grown from about 20 words per bullet through v1.48.2 to 176 at v1.58.0, carrying mechanism, benchmark figures and review history that the issues, merge requests and doc.go sections the same entries linked to already recorded. The v1.59.0 libc bullet spent about 300 words paraphrasing the Performance section that the last bullet of that entry announces. Rewrite the six sections from v1.55.0 to v1.59.1 to answer what changed, does it affect me and where do I read more, and stop there: 4233 words down to 1416. No link and no credit is dropped except for eight references to mechanism and review history that remain reachable from the issues still linked. HACKING.md gains an Entry style rule under CHANGELOG so the entries stay that way. The limit is sentences rather than words, one to three for an ordinary change and five at the outside, because ten of the twenty-two rewritten bullets still run past sixty words and a ceiling broken by the commit that introduces it is not a rule. No code and no shipped documentation changes, so this adds no entry of its own to the pending v1.59.1 section. Updates #258. Co-Authored-By:
-
cznic authored
Add LICENSE-3RD-PARTY.md, a full-text inventory of every third-party component this module carries, and licensegen/, the tool that generates it from the module graph, the license files in the module cache, the notice files dependencies carry, and the vendored C. Components are classified by what they actually do rather than by where they appear in go.mod: linked into the binary (the seven runtime modules plus SQLite 3.53.4, sqlite-vec v0.1.9 and the test_demovfs.c ancestor of vfs/), compiled into the tests only, or merely present in the module graph. The last group is where the only MPL-2.0 dependency anywhere in the graph sits, reached through modernc.org/libc's go.mod and in no import closure -- an SBOM built from the module graph will flag it, so the file says so plainly. musl libc, Go, go-netdb and NixOS/nixpkgs reach this module through modernc.org/libc's own LICENSE-3RD-PARTY.md, which is reproduced whole. Seventeen distinct license texts cover thirty-five components; where several share a text it appears once, with every copyright notice merged in place. The LICENSE name prefix is load-bearing. go mod vendor matches metadata files by case-sensitive prefix, so a name like 3RD_PARTY_LICENSES.md would never reach a downstream vendor/ tree -- the same trap that renamed SQLITE-LICENSE to LICENSE-SQLITE in v1.57.0. Output is deterministic and carries no timestamp; licgen -check fails on a stale file. Run make licenses after any dependency bump or re-vendoring. Co-Authored-By: -
cznic authored
Add a CycloneDX 1.6 and an SPDX 2.3 SBOM, generated by licensegen from the same inventory the license document is built from, and SBOM.md explaining what they cover and what they deliberately do not claim. Generating them here rather than with cyclonedx-gomod or syft is the whole point. Those tools read the module graph, and the largest component this module ships is not in it: lib/ is SQLite 3.53.4, C transpiled to Go by modernc.org/ccgo and committed here, with no go.mod entry and no name anywhere in the graph. Same for sqlite-vec in vec/ and the VFS bridge in vfs/. A consumer asking whether this project ships a version of SQLite with a CVE against it gets a wrong answer from a module-graph SBOM, not a cautious one. Both documents name it, version it and carry a pedigree note saying where it came from. The same holds in the other direction: modernc.org/libc carries notices for upstreams vendored into it, musl libc among them, wh...
-
cznic authored
Add SECURITY.md and point GOVERNANCE.md at it, replacing the two-sentence security line it carried. Three private reporting channels, so that nobody has to open an account they do not want: GitHub private vulnerability reporting, which is enabled on the mirror; a confidential GitLab issue on the canonical repository; or the project's GitLab Service Desk address, whose tickets are confidential by default because the project is public. No personal address is published. What is promised is what two maintainers can actually deliver: a reply aimed at seven days, and no fix deadline. Where the flaw sits decides what a fix costs -- driver code is ours to change today, while a fault in the transpiled C may mean re-vendoring and a coordinated modernc.org/libc release. Only the latest release is supported; there are no maintenance branches and never have been. Scope names the bug class that is peculiar to this project: a transpilation fault, where the generated Go in lib/, vec/ or vfs/ does not faithfully implement the C it was produced from. Those are ours, not upstream's. A flaw in SQLite's own code reaches users through a release of this module rather than through their system packages, so it should be reported here as well as upstream. Disclosure runs through a GitHub Security Advisory and then through the Go vulnerability database, so that govulncheck reports it to every user of the module. That last step is treated as part of shipping the fix rather than as an optional extra: this module has no entry in that database today, and an advisory nobody's tooling reads is not a disclosure. Co-Authored-By: -
cznic authored
Contribution guidance existed only in GOVERNANCE.md and HACKING.md, neither of which a first-time contributor is likely to open, and HACKING.md is about the maintainer side: landing merge requests, the CHANGELOG, tagging. CONTRIBUTING.md covers where to send a change -- a GitLab merge request, or a GitHub pull request that gets cross-merged by hand -- and leads with the thing that is not visible from the tree: most of the Go here is generated from C by modernc.org/ccgo, and edits to it are silently lost at the next re-vendoring. A table says which paths are generated and which of the files sitting beside them are hand-written and safe to edit, since lib/ holds both. It also covers building and testing, the absence of CI on merge requests, the libc pinning constraint, the AUTHORS versus CONTRIBUTORS distinction, and that contributions are taken under BSD-3-Clause with no CLA. While writing it: this module supports 20 GOOS/GOARCH ...
-
cznic authored
LICENSE-3RD-PARTY.md, SBOM.md, sbom.cdx.json and sbom.spdx.json are generated from the module graph and from the vendored C. A dependency bump or a re-vendoring makes all four wrong at once, and nothing in the repository would have noticed. licgen -check rebuilds them in memory and fails if what is committed differs; this wires that into a pipeline. The job runs only when something that feeds those documents changes: go.mod, go.sum, licensegen/, lib/sqlite.go and vec/vec.go, which carry the SQLite and sqlite-vec version strings, the upstream licence files, and the four documents themselves. Ordinary commits cost no CI minutes. The workflow rules keep it to one pipeline per change rather than a branch pipeline and a merge request pipeline both. This is the only CI job here, and it is not a substitute for anything: tests and cross-platform builds still run on the modernc.org/builder farm, as HACKING.md describes. CONTRIBUTING.md said there was no CI at all, which this makes untrue, so it and CLAUDE.md are corrected in the same commit. Rehearsed against an empty module cache, which is what a fresh container has: 185 MB downloaded, the check completes in eleven seconds, go.mod and go.sum are untouched, and a deliberately stale document exits non-zero with a message naming it. The configuration passes GitLab's CI lint with no errors and no warnings. Co-Authored-By: -
cznic authored
SECURITY.md says how to report and what to expect. This is what happens afterwards: triage, scoping, the fix, disclosure, and the two cases a generic plan does not cover. Scoping is by layer, because the three layers this module is built from are not equally ours to fix. Hand-written Go is fixed and released here. Generated Go is not: the fix belongs in modernc.org/libsqlite3, which owns the transpilation and the patch set, and arrives here through make vendor. A flaw in SQLite's own C is upstream's to fix and ours to ship, with the v1.56.0 journal-rollback patch as precedent for carrying a local patch until upstream lands theirs. Four rules are written down because they are the ones that get broken under time pressure: never merge a pull request on the GitHub mirror, which breaks the mirror irrecoverably; never bump modernc.org/libc alone; tag manually and only when the builders are green; run make sbom after any dependency change. The part worth having before it is needed: a published Go module version cannot be recalled. Deleting the tag does not remove it from the mirror, and neither does deleting the repository. The remedy is a fixed release plus a retract directive, which is advisory by design. The retract block in go.mod already carries seven entries. Account compromise assumes the worst ordering, revoke first and diagnose second, and treats any release made during the window under the retraction path. The credential inventory and revocation steps are deliberately not in this file; they stay in the maintainers' private notes, as Bootstrap's plan does. Structure follows the two plans the module's own references point at, Bootstrap's and gitoxide's. Co-Authored-By: -
cznic authored
The guide said a generated file starts with "// Code generated for <goos>/<goarch> by 'generator ...', DO NOT EDIT." and that the tooling keys on that. Only 19 of the 271 generated files in lib/ say so. The other 252 carry the modernc.org/undup marker, and vfs/ carries the ccgo command line. A contributor grepping for the sentence as written would find 19 files and conclude the rest were safe to edit. What every generated file carries is Go's standard "// Code generated ... DO NOT EDIT." line, so describe that, and give the grep that lists the hand-written files. It prints exactly the six the table names. Reported-by: Ian Chechin Co-Authored-By: -
cznic authored
The in-scope list named DSN parsing, connection hooks, user-defined functions, virtual tables, the VFS bridge and file locking, but not pcache/: hand-written Go that SQLite calls through trampolines for every page it reads. It is the least visible part of the driver and the one where a fault -- a page returned for the wrong key, or freed while SQLite still holds it -- is silent corruption rather than a crash. Reported-by: Ian Chechin Co-Authored-By: -
cznic authored
Phase 2 asked whether a fix moves modernc.org/libc, but the order that follows was written down nowhere, and under time pressure the order is what people get wrong. It is four releases, each pinning the one before: libc, libsqlite3, libsqlite_vec, then make vendor here and the builders before the tag, with lib/ and vec/ never on different libc versions. Phase 6 now singles out the mirror push credential: it can change what GitHub users see while GitLab looks fine, so nothing on the canonical side reveals its misuse. A builder result that cannot be trusted is now an incident. The dashboard gates tagging, so a false green is a release decision made on bad data. Also corrected: the Phase 3 table sent generated vfs/ code through libsqlite3. Its C is vfs/c/vfs.c in this repository, transpiled here, as CONTRIBUTING.md already says. pcache/ joins the hand-written row. CLAUDE.md's summaries follow. Reported-by: Ian Chechin Co-Authored-By: -
cznic authored
Driver.Open documented each _pragma value as "a PRAGMA statement". It is run as SQL text with PRAGMA prepended, and anything after a ';' runs too: "_pragma=foreign_keys(1);ATTACH 'x.db' AS x" attaches, and creates, x.db. The documentation now says so. StrictPragmas(true), opt-in and off by default, makes a connection reject a _pragma value holding more than one statement, with an error wrapping ErrMultiStatementPragma. It is process-wide, like OFDLocking, and deliberately not a DSN key: the DSN is what it guards, and whoever writes one could leave the key out. It is recommended for any application whose DSN is not a compile-time constant. The check runs in the validation phase, so a rejected DSN applies nothing. It is lexical and compiles nothing: preparing the text would compile the statements it exists to reject, some PRAGMAs take effect at compile time, and it would run before busy_timeout is set. It follows SQLite's tokenizer rules for quotes and comments. TestSingleStatementAgainstSQLite and FuzzSingleStatement hold it to SQLite's own parser. Fuzzing found three disagreements, all in the harmless direction, and the check now matches SQLite on each: a /* at the very end is a slash, not a comment; SQLite stops at a NUL; and \v counts as whitespace only inside a run of other whitespace, so the check rejects it outright. The three inputs are kept as seeds. 5.1 million executions after the last fix found nothing. Found while reviewing a threat model of the DSN surface drafted by Ian Chechin. Co-Authored-By: -
cznic authored
A DSN names the file to open or create, selects the VFS, and can run SQL through _pragma. One taken from an untrusted source is untrusted SQL plus file system access, and nothing the driver could do would make it otherwise without removing what a DSN is for. Say so, the way the policy already says it for SQL injection. What stays in scope is the driver keeping its own promises about a DSN: a way around StrictPragmas, or around the rule that a rejected DSN applies nothing, is a bug of ours. Flaws in DSN parsing remain in the in-scope list. Co-Authored-By: -
cznic authored
Context cancellation calls conn.interrupt from the goroutine watching the context, while the query it interrupts may be running on c.tls. A libc.TLS is not safe for concurrent use, and c.Lock only guards against Close. This was safe only because the transpiled sqlite3_interrupt is a single atomic store that never touches the TLS, in every variant in lib/. Nothing states or tests that, and a future SQLite or ccgo that gives the function a local would turn a cancellation into silent memory corruption. Interrupt on a fresh TLS instead. Measured on linux/amd64, NewTLS plus Close is about 140 ns and 224 bytes, paid only when a query is actually interrupted; one TLS held per connection would cost every connection about half a kilobyte here, and a 4 KiB stack segment on the other targets, for nothing. A TLS's stack never shrinks, but it grows only through Alloc, which sqlite3_interrupt never calls. Found while reviewing a threat model of the connection lifecycle drafted by I...
-
cznic authored
Two properties of connections were documented nowhere. State set on a pooled connection -- PRAGMAs set with Exec, ATTACHed databases, temporary tables, anything registered through sql.Conn.Raw -- is inherited by the next caller to borrow it; the driver does not reset it. And a driver connection reached through Raw is not safe for concurrent use: SQLITE_OPEN_FULLMUTEX serializes access inside SQLite, but the per-connection libc.TLS is used before that mutex is reached. FunctionContext gains a maintainer comment. It has no methods, so a retained one is inert today; any method added must stay safe when called on a retained one, because by then it is zeroed or belongs to another invocation. Co-Authored-By: -
cznic authored
Fetch retires the stub it holds for a key whenever the Cache returns nil or a different Page for it. That is right for an unpinned key and a use-after-free for a pinned one: SQLite keeps reading and writing the stub's page after the binding has freed it, and the database is corrupted without any error. The Page contract forbids the Cache from doing that, but nothing checked. The binding sees every pin and unpin, so it now records which stubs SQLite holds -- set by Fetch, cleared by Unpin, dropped by Truncate and Destroy -- and panics on the violation instead of freeing the memory. A crash names the bug; silent corruption does not. modernc.org/sqlite/pcache keeps the contract and is unaffected. Cost: one bool per stub. The threading comment is corrected on the way: it said serialisation held because there is no shared-cache mode, but cache=shared in a file: DSN reaches SQLite, and callbacks then arrive from several goroutines, serialised by...
-
cznic authored
A program importing modernc.org/sqlite/vec cannot register a page cache: vec's init installs sqlite-vec with sqlite3_auto_extension, which initializes SQLite, SQLite refuses SQLITE_CONFIG_PCACHE2 afterwards, and Go runs vec's init before any code in a package importing it. This was written down only in a test comment. It is now on RegisterPageCache, in the vec package documentation and in the pcache package documentation. PageCache.Create may be called concurrently, since connections open on whichever goroutines open them. Cache's documentation claimed no concurrent calls because there is no shared-cache mode; under cache=shared the calls are still serialised, by SQLite, but from several goroutines, so an implementation wants a mutex to run clean under -race, as pcache's does. Cache now also states the pinned rule the binding enforces. CHANGELOG entries for this and the previous commit. Co-Authored-By: Claude Opus 5 (1M context) <...
-
cznic authored
No test opened a database through the pluggable page cache: vec_test.go pulls vec's init into the test binary, which initializes SQLite before a page cache can be registered. Built with -tags pcachepool, as make test_pcache does, vec_test.go drops out and pcachepool_test.go registers modernc.org/sqlite/pcache in an init, so every test in the suite runs every page through the binding. A test fails the build if the Pool is not in effect, so the mode cannot quietly test SQLite's own cache instead. The whole suite passes that way, and under -race (601 s, linux/amd64). CONTRIBUTING.md lists the target; the stale comment in pagecache_test.go that deferred an end-to-end test now points here. Co-Authored-By:
-
cznic authored
The page cache entry said pcache/ is hand-written Go that SQLite calls for every page it reads. The code SQLite calls is the binding, pagecache.go and pagecache_trampolines.go; pcache/ is the optional reference implementation; and none of it runs unless an application calls RegisterPageCache. The entry now says all three. "Hardening you can use today" listed two opt-ins; StrictPragmas is the third. CLAUDE.md's summaries follow. Co-Authored-By:
-
cznic authored
FS.Close unregistered the VFS, freed its sqlite3_vfs and dropped the file system's handle without asking whether anything was still open through it. SQLite keeps a pointer to the VFS in every connection, so the next query on a connection left open went _hasHotJournal -> vfsAccess, read pAppData out of the freed struct and panicked -- or, had the memory been reused by the next New, resolved to a different, live file system. Each registered file system now counts the files open through it, and Close returns an error wrapping the new ErrInUse while any is, leaving the VFS registered. The documentation says to close the databases first and not to call Close concurrently with an open, a window no count can close: SQLite finds the VFS under its own mutex and calls xOpen outside it. One uintptr counter numbers file systems and files alike. On the 32-bit targets it wraps after 2^32 opens, and addObject then overwrote whatever held the ha...
-
Ian Chechin authoredmake vendor copies transpiled Go from sibling checkouts of libsqlite3 and libsqlite_vec, and nothing recorded which revisions produced the lib/ and vec/ a release carries. It now starts by refusing a dirty checkout, two checkouts on different libc versions, or a libsqlite_vec built against another libsqlite3 commit, and ends, after the cross-builds, by writing vendor.json: the two commits, what their go.mod files require, the Go toolchain, the undup pin and a digest of the files it produced. internal/vendorstamp checks the stamp against go.mod and the files on disk. The suite runs that check, so the builders fail on a dirty or stale stamp, and a new CI job runs it alone. The toolchain is recorded because it is an input: vendoring from v1.14.5 and v0.5.0 reproduces master's lib/ byte for byte, and vec/ only under Go 1.27, whose gofmt keeps two lone // comment lines that 1.22 to 1.26 drop. The committed vendor.json comes from a ful...
-
Ian Chechin authored
-
Ian Chechin authored
make vendor VENDORFLAGS=-allow-dirty passes the flag to -preflight and -stamp, for a debug build from a modified libsqlite3 checkout as doc.go describes. The stamp then records the checkout as dirty, so the last step of make vendor and TestVendorStamp fail, by design. IRP.md and the CHANGELOG entry name all three refusals and say that a libc bump now goes in the same push as make vendor. /vendor is ignored, since a refused make vendor leaves the tool behind. The digest error says how to restore a tree expanded with undup -expand.
-
-
cznic authored
lib/seh.go implements modernc_seh_try() and modernc_seh_inject(), the two externs that internal/sqlite_issue221.patch in modernc.org/libsqlite3 (branch seh-emulation) makes wal.c call instead of using __try/__except: the protected body runs under debug.SetPanicOnFault() with a recover() guard, so the EXCEPTION_IN_PAGE_ERROR that today kills the process ("unexpected fault address ... signal 0xc0000006" in #221) - and a SIGBUS on unix - becomes SQLite's own SQLITE_IOERR_IN_PAGE (8714) after walHandleException(), with the connection staying usable. Only faults inside the wal-index pages are accepted; anything else still crashes. SehInject(n) simulates a fault at the n-th SEH_INJECT_FAULT site for tests. seh_test.go covers it through database/sql (a real fault from a truncated -shm file, and one simulated fault per reachable site for reads, writes and checkpoints) and skips itself while the vendored lib/ predates the patch, which it does until lib/ is re-vendored from a libsqlite3 that carries it. The CHANGELOG entry is a placeholder to be dated at release time. Nothing changes when no fault occurs; the guard costs ~14 ns per protected WAL entry point. See HANDOFF-seh-emulation.md on the libsqlite3 branch. Co-Authored-By: -
cznic authored
- lib/: `make vendor` from ../libsqlite3 at v1.15.0 (879cf166). SQLite stays 3.53.4. Every target now calls _modernc_seh_try (1 site) and _modernc_seh_inject (17 sites), defined in the hand-written lib/seh.go, so a fault on the memory-mapped -shm file fails the statement with SQLITE_IOERR_IN_PAGE instead of crashing the process (#221). vec/ is unchanged: ../libsqlite_vec is still at v0.5.0. - The lib constants are evaluated as C expressions now (cc v4.29.4): on linux/amd64 42 change value (WALINDEX_PGSZ is 32768, not 0; HASHTABLE_NPAGE_ONE 4062, ROWSET_ENTRY_PER_CHUNK 42), 33 appear and nine that never had a meaningful value are gone (INFINITY, MB_CUR_MAX, NAN, RESERVED_BYTE, SHARED_FIRST, SQLITE_CANTOPEN_BKPT, SQLITE_CORRUPT_BKPT, SQLITE_DEFAULT_LOOKASIDE, SQLITE_MISUSE_BKPT). - vendor_libs/main.go: skip SQLITE_STATIC like SQLITE_TRANSIENT; both are typed constants in lib/defs.go and the transpile now emits SQLITE_STATIC. - go.mod: modernc.org/libc v1.77.1, golang.org/x/sys v0.48.0, go 1.26.0; `make sbom` regenerated the four license/SBOM documents. - CHANGELOG.md: two bullets in the pending section. Verified on linux/amd64: make vendor (build_all_targets included), make editor, go vet ./lib (no duplicate declarations), go test -run TestSEH, go test . ./vec/... ./vfs/... ./pcache/... Co-Authored-By:
-
cznic authored
Co-Authored-By: -
cznic authored
make vendor with ../libsqlite3 at v1.15.0 (879cf166) and ../libsqlite_vec at v0.6.0 (fa0b5d4), both clean, under go1.27.0. lib/ is byte-identical to the re-vendor in 2e433240 and vec/ to v1.59.0: libsqlite_vec's transpiles did not change between v0.5.0 and v0.6.0, only its go.mod and autogen snapshots did. make sbom leaves the four license and SBOM documents unchanged; licgen -check and TestVendorStamp pass; every target compiles one _modernc_seh_try and 17 _modernc_seh_inject call sites. Co-Authored-By:
-
cznic authored
The pending section becomes v1.60.0: Go 1.26 is required, the libc pin moves to v1.77.1, lib gains SehInject and SehPending and loses nine constants, so this is a minor bump on v1.59.0. The number and the date are the maintainer's call at tag time, per HACKING.md. Co-Authored-By: -
cznic authored
1071 words in 17 bullets become 730 in 14, per the HACKING.md entry-style rule: the documentation-only bullets stop paraphrasing the documents they announce, SECURITY.md joins IRP.md and LICENSE-3RD-PARTY.md joins the SBOM, the stranded "Resolves #257" line attaches to the URI-parameters bullet it credits, and the vendor.json bullet drops "the vendored code is unchanged", which was true of merge request !140 alone and false of this release, whose lib/ is re-vendored. The release-critical and behavior-change bullets are unchanged. Co-Authored-By: