sqlite: add SQLITE_CONFIG_PCACHE2 wrapper (API + skeleton)

Ian Chechinrequested to merge
Deln0r/sqlite:pcache2-api-draft into master

Motivation

Issue #204 (closed) asks for a pluggable cross-connection page cache, and on 18 Mar 2025 you noted that the right shape is exposing SQLITE_CONFIG_PCACHE2 as a Go API rather than reimplementing pcache1 in pure Go. This draft MR proposes that public API.

The MR is intentionally a skeleton. It lands the API surface, the SQLITE_CONFIG_PCACHE2 wiring, and tests. The production pool-backed implementation and the memory-utilization benchmark that demonstrates the reduction are deferred to follow-up MRs so the API shape can be reviewed independently of the implementation.

What this MR delivers

  • pagecache.go: public PageCacheModule struct, Page / PageEq interfaces, RegisterPageCacheModule and MustRegisterPageCacheModule functions, typed sentinel errors ErrPageCacheTooLate and ErrPageCacheConflict.
  • Per-arch type alias shim (two build-tagged files, ~5 lines each). Today the cznic-emitted struct name is Tsqlite3_pcache_methods2 on most arches and Sqlite3_pcache_methods2 on freebsd_386, freebsd_arm, and netbsd_amd64. The shim resolves this without touching the generated code.
  • Small touch to conn.go (~4 lines wrapping openV2 in withOpenGate): take pcacheState.openGate.RLock, mark the opened flag with an atomic Store, then call sqlite3_open_v2.
  • pagecache_test.go: layout assertion (asserts the offset relationships we depend on plus the total struct size), misuse tests (nil module, missing required func, conflict, same-pointer idempotency, post-open registration), and a gate-RWMutex smoke test.

What this MR explicitly does NOT deliver

  • The pool-backed production cache. Deferred to MR-B.
  • The memory-utilization benchmark for #204 (closed). Belongs in MR-B alongside the production impl so the benchmark exercises the real code path.
  • Cross-connection sharing wire-up (driver-level shared cache plumbing). Deferred to MR-C.
  • Lifecycle and error-recovery hardening for the production impl. Deferred to MR-D.
  • Per-connection override / DSN flag. Deferred to MR-E.

Three issues this MR addresses

1. Lifecycle race between RegisterPageCacheModule and the first Open

SQLITE_CONFIG_PCACHE2 must be installed before sqlite3_initialize, which fires implicitly inside the first sqlite3_open_v2. Without ordering, concurrent Register and Open can leave the engine partially initialized with the default pcache1 while the binding believes the override took effect.

Resolution in this MR: pcacheState.openGate sync.RWMutex plus atomic.Bool for the opened flag plus sync.Once wrapping the Xsqlite3_config call. The atomic.Bool is for the open hot path (Store under RLock, no Lock cost per open). Register takes the write lock, which drains all in-flight Opens, then reads the flag and proceeds with Xsqlite3_config under sync.Once. The once body uses a defer / recover guard so a panic during populate or config rolls back the half-set state. Trade-off: Register may block for the duration of any in-flight Open (WAL recovery, cold-file-lock contention).

2. Stub identity will be owned by the binding, not the impl

If the binding minted a fresh sqlite3_pcache_page stub on every Fetch and keyed identity off Page.Buf(), a discard=true Unpin between fetches would free the stub while SQLite still held its address in PgHdr->pPage (set by _pcacheFetchFinishWithInit). Use-after-free, undetectable by integrity_check.

This MR exposes the Page interface as a stable surface for impl authors but does not yet route Fetch through it; the production impl in MR-B will own the sqlite3_pcache_page stub via a per-cache byKey / byStub table and signal eviction via Unpin(p, discard=true), Truncate, or Destroy. A debug invariant under the pcachedebug build tag or under -race will re-sample Buf() and Extra() and panic on drift; un-tagged release builds will pay nothing.

3. Arch-portable wiring

A naive implementation using hardcoded amd64 byte offsets (16, 24, ..., 96) would corrupt the struct on every 32-bit arch: lib/sqlite_linux_386.go uses 8, 12, ..., 48 because uintptr is 4 bytes and there is no padding after the int32 FiVersion. lib/sqlite_netbsd_amd64.go has an explicit F__ccgo_pad1 [4]byte. The MR uses named-field writes only (FiVersion, FpArg, FxInit, ...), allocation via libc.Xcalloc matching vtab.go:130-141, and TestPCacheMethods2Layout asserts the field offset relationships plus the total struct size, so a future regeneration drift fails CI.

Sub-MR plan

Sub-MR Scope When
MR-A API + skeleton (this draft) now
MR-B Pool-backed production cache + memory-utilization benchmark after MR-A merges
MR-C Cross-connection sharing wire-up after MR-B merges
MR-D Lifecycle and error-recovery hardening after MR-C merges
MR-E Per-conn override (DSN flag plus opt-in API) after MR-D merges

Updates #204 on this MR; Closes #204 reserved for the MR that lands the benchmark.

Lifetime notes

  • The pcacheMethods2 C struct is allocated once via libc.Xcalloc and lives until process exit. Re-registration with a different module returns ErrPageCacheConflict; reload is not supported in this MR.
  • User Shutdown may never fire in practice. The binding does not call sqlite3_shutdown. An impl author should not put critical cleanup in Shutdown.
  • Init and Shutdown are serialized by the engine; all other callbacks may be invoked concurrently and must be threadsafe. Build assumes SQLITE_THREADSAFE=1.
  • Callbacks must not call RegisterPageCacheModule directly or transitively. Init in particular runs under the openGate read lock held by the Open path, so a re-entrant call would deadlock on the same gate's write lock.

Questions

  1. Is pointer-identity idempotency on duplicate RegisterPageCacheModule acceptable, or do you prefer fail-loud on every duplicate Register (vtab-module-name style)?
  2. On a Fetch contract violation (impl returns a different (Buf, Extra) for an un-evicted key with a live stub), do you want the binding to panic (A) or return NULL plus a one-time log (B)?
  3. Should PageEq remain in the public API (A) or stay private as an internal type assertion (B)?
  4. Type-alias shim approach for the three old-generator arches: accept here (A), or send a regenerate-old-arches patch first (B)?

Merge request reports