refactor: embed the Postgres driver from django-vpg instead of carrying it
Ready for review. django-vpg 0.3.0 is published to PyPI and !19 (merged) (the standalone wire-transaction-status fix) is merged, so this MR now targets main, pins vpg-pyo3/vpg-core to the released tag = "v0.3.0", and asks for django-vpg>=0.3.0 from PyPI. (History: it was drafted stacked on !19 (merged) against a pre-release git rev; both pins are now the released tag.)
What this does
Deletes gt_rust's copy of the Postgres driver — crates/gt-postgres, src/postgres/, python/gt_rust/dbapi.py, python/gt_rust/django_backend/ — and links django-vpg instead: vpg-pyo3 as an rlib, its pyclasses re-exported into our own _rust module. Same embedding shape gt_rust already uses for vcache's Valkey driver.
One .so, one pool, one runtime. vpg_core::set_runtime_provider(gt_runtime::get_runtime) runs before vpg registers anything, so the pool it builds rides gt_rust's fork-safe runtime instead of starting a second one, and the ingest pipeline keeps sharing that pool.
Net: the whole in-tree driver removed (~17k deletions) in favour of the embedded package.
Two things the work turned up
1. PyInit__driver collides between django-v* packages. The symbol is PyInit_<module basename>, and both vcache-pyo3 and vpg-pyo3 called their module _driver — so linking both rlibs into one cdylib is a duplicate-symbol link error. Fixed on the vpg side (django-vpg!2 (merged)) two ways: vpg's module is renamed to django_vpg._vpg_driver, which is what actually guarantees the link, plus a default-on standalone-module feature so embedders don't compile an entry point they never use. The feature alone would not have been enough — Cargo features unify across a dependency graph, so anything else depending on vpg-pyo3 with default features would silently re-enable it. Worth knowing for django-vtasks or any future family member that grows a Rust extension; note gt_rust's .so still exports vcache's PyInit__driver, harmlessly, since only one of the two needed to move.
2. gt_rust.dbapi has to claim sys.modules["django_vpg._vpg_driver"] before django_vpg is imported. django_vpg.dbapi imports its own extension by name, and pyo3 statics are per-.so: without the alias the process quietly gets two pools, two runtimes, and a pinned session in one invisible to the other. The shim raises rather than proceed if something already imported the standalone module.
gt_rust.dbapi and gt_rust.django_backend remain as aliases, so DATABASE_ENGINE = "gt_rust.django_backend" keeps working — adopting vpg needs no coordinated glitchtip-backend deploy, and rollback doesn't touch settings.
Tests
The driver's own suites (dbapi, Django backend, async backend, asyncio driver) moved to django-vpg with the code. What this repo owns now is tests/test_vpg_embedding.py, which asserts the things that only break here:
django_vpg._vpg_driveris our module, not vpg's own extension- the classes are the same objects, not same-named twins from a second
.so gt_rust.django_backend/.basestill resolve (whatDATABASE_ENGINEloads)- the namespace mirror includes the underscore helpers the backend imports by name
- live: connections opened through
gt_rust.dbapiand throughdjango_vpg.dbapishare one driver
tests/test_ingest_session.py grew its own _connect helper, having borrowed one from a test module that now lives in vpg.
Docs
AGENTS.md's surface list and exposure table point at django-vpg and say plainly that driver changes go there while the embedding stays here. The session/pool contract is now a pointer to vpg's README rather than a second copy that can drift. The vendor gotcha records that vpg carries a byte-identical copy of the tokio-postgres tree, that [patch.crates-io] only applies from the top-level workspace (ours wins for the whole graph, vpg's goes inert when embedded), and that the two must be mirrored in one pass. docs/postgres-driver-integration.md is marked as history — its "single pool + one runtime" goal is exactly what this preserves.
Verification (against the published 0.2.0)
./scripts/check.shgreen (fmt, clippy-D warnings, tests for the extension and the four remaining core crates).- Built with
maturin develop --releaseon CPython 3.14:vpg-pyo3 v0.2.0+vcache-pyo3 v3.1.0rlibs link into one cdylib with no duplicate-symbol error — the_vpg_driverrename holds. - Python suite green against live PostgreSQL 18 with
django-vpg==0.2.0installed from PyPI (not an editable checkout): 35 passed, 15 skipped (valkey-only), including the live shared-pool test that opens a connection through bothgt_rust.dbapianddjango_vpg.dbapiand asserts one driver. - Confirmed the vendored tokio-postgres patch applies across the graph: vpg-core compiles against our patched copy, which is what makes
Client::transaction_status()resolve.
Update — re-pinned to v0.3.0
django-vpg 0.3.0 shipped (param-binding punning fixes, unified SQL lexer + per-statement template cache, DECLARE failure rollback; A/B-benchmarked +6% rps / −5% CPU on the drop-in sync path). This branch now pins tag = "v0.3.0" / django-vpg>=0.3.0 and re-verified: ./scripts/check.sh green, Python suite 35 passed / 15 skipped (valkey-only) against live PostgreSQL 18 with django-vpg==0.3.0 from PyPI, including the live shared-pool test. Vendored tokio-postgres trees confirmed byte-identical between the repos.