feat(postgres): COPY ... FROM STDIN with psycopg-shaped write_row

Why

glitchtip-backend is switching its payload-bearing ingest batch writes (issue events, log events) from composed INSERT statements to COPY ... FROM STDIN on both database drivers: COPY streams the batch body in small frames, so connection buffers stay bounded regardless of batch size, Python never composes the batch into one giant SQL string, and the server skips parse/bind of multi-megabyte statements — efficiency postgres itself can't buy back by scaling horizontally. The driver only supported the read direction (COPY TO STDOUT, used by cold storage); this adds the write direction with psycopg-shaped semantics so backend code is driver-agnostic.

What

• gt-postgres core (crates/gt-postgres/src/copy_in.rs): session-aware start (dedicated pool checkout in autocommit — matching how every other autocommit statement runs — or the pinned connection inside a transaction, same routing as COPY TO), a chunk sink over tokio-postgres CopyInSink, finish returning the CommandComplete row count, and abort-on-drop (CopyFail: the server discards every row and the connection returns to the pool usable).
• Wire tests: the scripted fake-PG server learned the COPY IN protocol (CopyInResponse, CopyData capture, CopyDone/CopyFail); tests cover the happy path + row count, abort leaving the pooled connection reusable, and SQLSTATE classification at COPY end (unique violation → Integrity).
• pyo3 wrapper (src/postgres/copy_in.rs): RustCopyIn rides the existing RustAwaitable machinery via a RawResult::CopyIn variant; async write/finish plus _sync twins. The pool checkout happens inside the runtime task, so a saturated pool never blocks the calling event loop.
• dbapi: cursor.copy() dispatches on direction — FROM STDIN returns a writable context manager with psycopg's write_row(), encoding rows to COPY text format (None, bool, int/IntEnum, float, str, UUID, datetime, dict, psycopg Jsonb-shaped wrappers (duck-typed — no psycopg import), list → array literal) and streaming in bounded 64 KiB chunks. The async cursor gains the matching async context manager; exiting with an exception aborts the COPY.

Testing

• ./scripts/check.sh green (fmt, clippy, cargo tests incl. 3 new wire tests).
• pytest suite green with live postgres (188 tests; new: encoder unit tests incl. escaping/arrays/enums, sync + async roundtrips, abort discards rows and the connection stays usable, duplicate PK surfaces as IntegrityError, COPY inside a transaction sees uncommitted DDL).
• glitchtip-backend's ingest + log test modules (82 tests) pass under DATABASE_ENGINE=gt_rust.django_backend with a wheel from this branch and the backend's COPY write path enabled for both drivers.

---

Written with AI assistance (Claude); human review required.

json/jsonb decode contract (second commit)

Reviewing the COPY tests surfaced a decode question, resolved against Django's actual psycopg adapter configuration (verified in django/db/backends/postgresql/psycopg_any.py): Django registers a TextLoader for jsonb — JSONField applies its own decoder, and a dict there raises TypeError — while plain json (what row_to_json()/json_agg() return in raw SQL) has no TextLoader and parses to Python objects. The driver returned raw text for both, and the query_batch value path inconsistently json.loads'd jsonb.

Now aligned on both paths: jsonb → raw text everywhere, plain json → parsed objects (serde_json AST built directly into Python structures). Pinned by a live contract test. glitchtip-backend's ingest, logs, and issue-events modules (206 tests) pass under DATABASE_ENGINE=gt_rust.django_backend with a wheel from this branch.

Structured server diagnostics (third commit)

COPY cannot express ON CONFLICT, so glitchtip-backend's ingest wraps it in a fallback that must distinguish a unique violation (retry via conflict-tolerant INSERT) from every other IntegrityError (missing partition, FK — doomed to fail the retry identically). Server errors previously crossed the Rust→Python boundary as one message string: the [SQLSTATE] prefix plus DETAIL/HINT/CONSTRAINT as appended text, with the remaining ErrorResponse fields (schema, table, column, positions, source location) dropped. Branching on the exact error meant parsing messages, and operators lost fields psycopg users get.

Now the full diagnostics cross structurally, psycopg-shaped end to end:

• core: PgDiag captures every DbError field under psycopg's Diagnostics attribute names; RawResult::Error / classify_pg_error carry Option<Box<PgDiag>> (None = never reached the server).
• pyo3: the fields ride the raised exception as a pg_diag dict. The COPY OUT start/read paths built their PyErr inside py.detach (no GIL → no attribute attach); they now carry PgErrParts out of the closure and build the exception GIL-side.
• dbapi: every DB-API Error exposes psycopg-parity .sqlstate and .diag, so driver-agnostic code — getattr(exc.__cause__, "sqlstate", None) == "23505" under Django — works unchanged on either driver, and keeps working when psycopg is eventually dropped.

Tested: scripted wire-server ErrorResponse with all 16 fields → PgDiag (and diag through COPY finish), translator unit tests, live duplicate-key tests asserting sqlstate/constraint_name via execute, sync COPY, and async COPY. glitchtip-backend's narrowed COPY fallback (!2449) runs green under DATABASE_ENGINE=gt_rust.django_backend with a wheel from this branch (297 tests).

Rust batch encoder: write_rows() (101dc995)

Measuring client CPU per batch (time.process_time, interleaved in-process A/B — wall time hides driver CPU behind WAL-fsync waits both drivers share) showed the Python COPY encoder burning ~3x psycopg's C Transformer on flat rows. psycopg is fast because its C code reads Python values through the CPython C API without ever re-entering Python; an earlier bulk-encode attempt here lost precisely because its conversion ladder called back into Python per value.

encode_copy_chunk + write_rows() apply that lesson — no Python call to format any value:

• exact-type pointer dispatch, with a cold subclass ladder mirroring the Python encoder (IntEnum, datetime subclasses, Jsonb/Json duck-typing, identical TypeError);
• PyDateTime C-API field access, timezone.utc pointer compare;
• uuid.UUID.int → u128, hex-formatted in Rust;
• borrowed &str with one escape scan (byte writers + escaping live in gt-postgres/src/copy_text.rs with cargo tests, including hostile column/row-injection bytes);
• dict/Jsonb via one shared dumps per value (dbapi._copy_json_dumps: orjson with OPT_NON_STR_KEYS when importable — TypeError falls back to stdlib json so >64-bit ints in user payloads can't fail a batch — else stdlib json; both the Python and Rust encoders use the same callable, one dialect).

The Python context managers stream the encoded batch in ~64 KiB chunks (one Python↔️Rust crossing per chunk, GIL held per chunk). Output is byte-identical to write_row() for every deterministic type (parity matrix test incl. namedtuples, ±2^127/2^200 ints, year<1000, hostile strings); floats/JSON are value-identical.

CPU per batch vs psycopg (interleaved medians, localhost PG17): 12-col log rows n=1000: 1.33 vs 1.42 ms; 13-col event rows with JSON payloads n=1000: 4.63 vs 12.11 ms; n=5 batches at parity. Independently reviewed (fresh-context security + correctness passes); findings folded in: the orjson big-int fallback, list/tuple-subclass array parity, min_bytes=0 guard.

C-API value marshalling: uuid/datetime cells and params (5193a183)

The same no-Python-re-entry rule applied to the two remaining per-value crossings, measured on both wall and CPU (interleaved medians vs psycopg, localhost PG17):

• UUID read cells — previously Uuid::to_string() + UUID(str), whose __init__ re-parses the hex in pure Python; the single most expensive cell type GlitchTip decodes (issue/event/span ids). Now built the way psycopg's C loader does it: UUID.__new__(UUID) + generic setattr of the int/is_safe slots from the wire u128. 2000 rows × 4 uuid cells: wall 23.5→13.8 ms, CPU 12.2→3.9 ms (from 1.4x/1.5x behind psycopg to 0.84x/0.56x). Object fidelity pinned by tests: type, eq, hash, str, version, is_safe, immutability, pickle.
• datetime/date/uuid params — previously isoformat()/str() followed by a chrono/hex re-parse of the text. Now field reads via the PyDateTime C API and uuid.int → Uuid::from_u128, scalars and array elements both; non-UTC zones keep the isoformat fallback. Exact-type fast paths (str/bool/int/float) ahead of the classification ladder. unnest 3×1000 arrays: wall 5.3→2.8 ms, CPU 2.7→0.7 ms (0.25x/0.09x of psycopg).

Two deliberate behavior notes: lone-surrogate str params now raise UnicodeEncodeError like psycopg (previously silently sent U+FFFD-replaced text), and a datetime inside a date[] array still raises ValueError (field access would have silently truncated it — guarded + tested). Independently reviewed (fresh-context pass over the unsafe construction, UUID fidelity, datetime/fold/timezone parity, and ladder reordering).

Lossless NUMERIC params + immediate result release (54c7af68)

• NUMERIC params match psycopg exactly. The former rust_decimal binding silently rounded Decimals past ~28 significant digits and rejected NaN/±Infinity. Params now encode PG's binary NUMERIC wire form straight from str(Decimal) (encode_numeric_text, the cargo-tested mirror of the PgNumericText read codec): arbitrary precision, display scale preserved, specials mapped like psycopg's dumper. Fuzzed against psycopg with ~10k adversarial Decimals in review — zero mismatches. Hardening from that review: huge exponents (Decimal("1E+999999999999999999")) are a clean DataError instead of an unbounded allocation, float4/float8 targets reject out-of-range finite values instead of saturating to Infinity, scientific notation expands to plain digits for text targets, and non-numeric inferred targets refuse loudly.
• Awaited result sets free immediately. The awaitable used to keep its own reference to a delivered result, pinning a dropped batch until the coroutine next yielded. Plain-await delivery now moves the value to the awaiter; multi-consumer futures (e.g. gather(aw) racing a plain await) are detected via the done-callback count and keep the value, and callback exceptions surface through loop.call_exception_handler instead of being swallowed. One deliberate asyncio.Future deviation: result() after a sole plain await raises InvalidStateError (loud) rather than returning the value again.