Commit 4552e53e authored by cznic's avatar cznic
Browse files

Merge branch 'vendor-stamp' into 'master'

vendor_libs, Makefile: record where lib/ and vec/ came from in vendor.json

See merge request !140
parents 3c3e178f 37751773
Loading
Loading
Loading
Loading
+2 −0
Changes for .gitignore: 2 added lines, 0 removed lines.
Original line number Diff line number Diff line
.claude/
/.vendor-preflight.json
/vendor
+37 −7
Changes for .gitlab-ci.yml: 37 added lines, 7 removed lines.
Original line number Diff line number Diff line
# The only CI this repository has, and deliberately small.
#
# Cross-platform building and the test suite run on the modernc.org/builder
# farm, not here; see HACKING.md. This pipeline exists for one thing that
# nothing else would notice: LICENSE-3RD-PARTY.md, SBOM.md, sbom.cdx.json and
# sbom.spdx.json are generated from the module graph and the vendored C, so a
# dependency bump or a re-vendoring silently makes all four wrong. licgen
# -check regenerates them in memory and fails if what is committed differs.
# farm, not here; see HACKING.md. This pipeline exists for two things that
# need no build of the transpiled code and fail fast:
#
# It runs only when something that feeds those documents changes, so ordinary
# commits cost no CI minutes.
# - LICENSE-3RD-PARTY.md, SBOM.md, sbom.cdx.json and sbom.spdx.json are
#   generated from the module graph and the vendored C, so a dependency bump or
#   a re-vendoring silently makes all four wrong. licgen -check regenerates them
#   in memory and fails if what is committed differs.
# - vendor.json records where lib/ and vec/ came from and a digest of them; see
#   internal/vendorstamp. It fails when a re-vendoring was not stamped, a
#   stamp says a sibling checkout was dirty, or generated files were edited by
#   hand. The test suite runs the same check, so the builders refuse such a
#   commit as well, and a release is tagged only when they are green.
#
# Each job runs only when something that feeds it changes, so ordinary commits
# cost no CI minutes.

stages:
  - check
@@ -61,3 +68,26 @@ generated-docs:
      fi
  after_script:
    - rm -f "$CI_PROJECT_DIR/licgen"

.vendor-stamp-inputs: &vendor-stamp-inputs
  - vendor.json
  - go.mod
  - lib/**/*
  - vec/**/*
  - LICENSE-SQLITE_VEC
  - internal/vendorstamp/**/*
  - vendor_libs/**/*
  - Makefile
  - .gitlab-ci.yml

vendor-stamp:
  stage: check
  image: golang:1.27
  interruptible: true
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes: *vendor-stamp-inputs
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes: *vendor-stamp-inputs
  script:
    - go test ./internal/vendorstamp/
+1 −0
Changes for CHANGELOG.md: 1 added line, 0 removed lines.
Original line number Diff line number Diff line
@@ -3,6 +3,7 @@
Entries for v1.38.1 through v1.44.1 and for v1.49.1 were added on 2026-09-05, reconstructed from the git history and the merge requests they cite; they were missing at release time.

 - 2026-09-15 v1.59.1:
     - `make vendor` now writes `vendor.json`: the `modernc.org/libsqlite3` and `modernc.org/libsqlite_vec` commits and the Go toolchain `lib/` and `vec/` were vendored with, so `git show vX.Y.Z:vendor.json` says which revisions a release carries. It refuses a dirty sibling checkout, siblings on different `modernc.org/libc` versions, or a `libsqlite_vec` built against another `libsqlite3`. The suite fails when `lib/`, `vec/` or the libc in `go.mod` no longer match the stamp, so a libc bump goes in the same push as `make vendor`. Tooling only; the vendored code is unchanged. See [GitLab merge request #140](https://gitlab.com/cznic/sqlite/-/merge_requests/140).
     - **`vfs.FS.Close` now refuses while a database opened through it is still open**, returning an error that wraps the new `vfs.ErrInUse` and leaving the VFS registered. It used to free the VFS the open connection still called through, so the next query crashed the process or read through freed memory. Close the databases first, then the `FS`.
     - Fix handle reuse in `modernc.org/sqlite/vfs` on 32-bit targets: after 2^32 file opens in one process the handle counter wrapped and could overwrite a live entry, such as a file system registered at start-up, and crash. 64-bit targets were not affected.
     - **The pluggable page cache now panics when a `Cache` breaks its contract** by returning nil, or a different `Page`, from `Fetch` for a page SQLite still holds pinned. It used to free memory SQLite was still using, corrupting the database without an error. Only a `Cache` implementation with that bug is affected; `modernc.org/sqlite/pcache` is not.
+2 −2
Changes for CLAUDE.md: 2 added lines, 2 removed lines.
Original line number Diff line number Diff line
@@ -16,8 +16,8 @@ The hand-written Go on top of that transpiled core implements the `database/sql/
- `vec/` — transpiled `sqlite-vec` v0.1.9, auto-registers via `sqlite3_auto_extension` in `patches.go` on package init. Activate by blank-importing: `_ "modernc.org/sqlite/vec"`. Covers the same 20 targets `lib/` does (20 is the count in `builder.json` and `make build_all_targets`; the 18 per-target files are fewer because `windows/amd64` and `windows/arm64` share `sqlite_windows.go`); `vec_test.go`'s `//go:build` constrains by GOOS only.
- `vfs/` — exposes a Go `fs.FS` as a read-only SQLite VFS. `vfs.New(fsys)` returns a registered VFS name; open with `?vfs=<name>`. C side is transpiled per platform from `vfs/c/vfs.c` via the `vfs/Makefile`.
- `vtab/` — Go-facing virtual-table API (no dependency on the transpiled C). `vtab.RegisterModule(db, name, module)` registers modules on **new connections only**; a nil `db` targets the driver registered as `sqlite`, a non-nil `db` the driver backing it (via `vtab.ModuleRegisterer`). The bridge to C lives in the top-level `vtab.go`. See `vtab/doc.go` for the contract (Updater/Renamer/Transactional optional interfaces, re-entrancy rules, ArgIndex/Omit semantics).
- `licensegen/` (own module, build tag `none`, built with `go build -tags none .`) — generates `LICENSE-3RD-PARTY.md`, `SBOM.md`, `sbom.cdx.json` (CycloneDX 1.6) and `sbom.spdx.json` (SPDX 2.3) from `go list -m all`, the license files in the module cache, the notice files dependencies carry (`modernc.org/libc`'s `LICENSE-3RD-PARTY.md`, which is how musl reaches us), and the vendored C. Classifies every component as linked / test-only / module-graph-only. Deterministic: no timestamps anywhere, SPDX's required `created` pinned to the epoch and its `documentNamespace` derived from the document's own content, so regenerating an unchanged tree is byte-identical and `./licgen -check` is meaningful. Both JSON documents validate against the published CycloneDX 1.6 and SPDX 2.3 schemas. `sbom.go` holds the SBOM writers, `main.go` the inventory and the Markdown. Invoked by `make licenses`, and wired into the repository's **only** CI job: `.gitlab-ci.yml` runs `licgen -check` when `go.mod`, `go.sum`, `licensegen/`, `lib/sqlite.go`, `vec/vec.go` or any of the four documents change. Nothing else runs in GitLab CI -- tests and cross-builds live on the builder farm. The `LICENSE` name prefix is load-bearing: `go mod vendor` matches metadata files by case-sensitive prefix, so `3RD_PARTY_LICENSES.md` would never reach downstream `vendor/` trees -- the same trap as the v1.57.0 `SQLITE-LICENSE` rename. Downloads into a scratch module so the repo's `go.sum` is never touched, and forces `GOWORK=off` so a `make work` workspace cannot leak into the document.
- `vendor_libs/main.go` (build tag `none`) — regeneration tool. Reads transpiled `ccgo_<goos>_<goarch>.go` from sibling repos `../libsqlite3` and `../libsqlite_vec`, rewrites package names and imports, and writes `lib/sqlite_*.go` / `vec/vec_*.go`. Invoked by `make vendor`.
- `licensegen/` (own module, build tag `none`, built with `go build -tags none .`) — generates `LICENSE-3RD-PARTY.md`, `SBOM.md`, `sbom.cdx.json` (CycloneDX 1.6) and `sbom.spdx.json` (SPDX 2.3) from `go list -m all`, the license files in the module cache, the notice files dependencies carry (`modernc.org/libc`'s `LICENSE-3RD-PARTY.md`, which is how musl reaches us), and the vendored C. Classifies every component as linked / test-only / module-graph-only. Deterministic: no timestamps anywhere, SPDX's required `created` pinned to the epoch and its `documentNamespace` derived from the document's own content, so regenerating an unchanged tree is byte-identical and `./licgen -check` is meaningful. Both JSON documents validate against the published CycloneDX 1.6 and SPDX 2.3 schemas. `sbom.go` holds the SBOM writers, `main.go` the inventory and the Markdown. Invoked by `make licenses`, and wired into one of the repository's two CI jobs: `.gitlab-ci.yml` runs `licgen -check` when `go.mod`, `go.sum`, `licensegen/`, `lib/sqlite.go`, `vec/vec.go` or any of the four documents change; the other runs `go test ./internal/vendorstamp/`. Nothing else runs in GitLab CI -- tests and cross-builds live on the builder farm. The `LICENSE` name prefix is load-bearing: `go mod vendor` matches metadata files by case-sensitive prefix, so `3RD_PARTY_LICENSES.md` would never reach downstream `vendor/` trees -- the same trap as the v1.57.0 `SQLITE-LICENSE` rename. Downloads into a scratch module so the repo's `go.sum` is never touched, and forces `GOWORK=off` so a `make work` workspace cannot leak into the document.
- `vendor_libs/main.go` (build tag `none`) — regeneration tool. Reads transpiled `ccgo_<goos>_<goarch>.go` from sibling repos `../libsqlite3` and `../libsqlite_vec`, rewrites package names and imports, and writes `lib/sqlite_*.go` / `vec/vec_*.go`. Invoked by `make vendor`. `vendor_libs/stamp.go` is its first and last step: `-preflight` refuses dirty or mismatched sibling checkouts before anything is touched, `-stamp` writes `vendor.json` (sibling commits, Go toolchain, undup pin, digest of the output) once the cross-builds have passed. `internal/vendorstamp` checks that file; `vendorstamp_test.go` runs the check in the suite, so the builders catch a stale or dirty stamp.
- `examples/` — runnable samples: `example1`, `connector`, `vtab_basic`, `vtab_csv`, `vtab_match`, `vtab_regexp`.
- `addport.go`, `issue198/`, `issue120.diff` — porting/regression scaffolding kept around for reference; not built.

+1 −0
Changes for CONTRIBUTING.md: 1 added line, 0 removed lines.
Original line number Diff line number Diff line
@@ -49,6 +49,7 @@ hand-written files.
| `lib/sqlite_*.go`, `lib/sqlite_g_*.go` | Generated from SQLite's C. Do not edit. |
| `vec/vec*.go` | Generated from `sqlite-vec`. Do not edit. |
| `vfs/vfs_*.go` | Generated from `vfs/c/vfs.c`. Edit the C, not the Go. |
| `vendor.json` | Written by `make vendor`: the sibling commits and Go toolchain `lib/` and `vec/` were vendored with, and a digest of them. Do not edit; the test suite fails when it does not match. |
| `lib/defs.go`, `lib/hooks*.go`, `lib/mutex.go`, `lib/libsqlite3_*.go` | Hand-written, no marker. Edit freely. |
| Everything at the top level, `vtab/`, `pcache/`, `examples/` | Hand-written. Edit freely. |

Loading