✨ feat(search): ONNX embeddings + hybrid RRF ranking (#2905 Phase 2)

Zax Shenrequested to merge
feat/search-rag-onnx-embeddings into dev

Phase 2 of GL !2905 — closes the search-first architecture work. Stacks on Phase 1 (FTS5 infra, !199 (merged) merged).

Three commits

cc35a9ba — Phase 2 main: ONNX embeddings + hybrid RRF + skill updates

  • Schema v4 migration adds 3 embedding tables (discussions_embeddings, audit_embeddings, file_registry_embeddings) with FK CASCADE where the source PK allows
  • NEW src/embeddings/{model,store,backfill}.ts — lazy-loaded @huggingface/transformers + bge-small-en-v1.5 (33MB, 384-dim); brute-force JS cosine over Float32 BLOBs; fire-and-forget backfill on server startup
  • 3 search tools gain mode='keyword'|'semantic'|'hybrid' (default hybrid) — RRF over FTS5 + cosine + recency-decay
  • Inline embed on every source write (discussion_append, audit_log, file_registry_*) — fire-and-forget post-write, never blocks
  • Skills (tmb_planning 193 / tmb_review 199 / tmb_recovery 131) get a one-paragraph search-first reference; all <200 LOC
  • Fallback: if onnxruntime-node fails to load, search degrades to FTS5-only with warning: 'semantic_unavailable' — no hard failure
  • 53 new test assertions across embeddings.test.ts (12) + search.test.ts extended (26) + schema-upgrade.test.ts extended (15)

ee12cc33 — hook chmod fix

scripts/hooks/pr-reviewer-spawn-prompt-shape.sh missed the +x bit during the slim-skills (!198 (merged)) Write-based conflict resolution. Caught by L0 install-smoke's test -x scripts/hooks/*.sh check. One-line chmod +x.

8427fcfb — defensive migrations + L0 Dockerfile auto-tracking

Two L0-discovered bugs:

  1. migrateV2toV3 (Phase 1, !199 (merged)) and migrateV3toV4 (Phase 2) referenced base tables in CREATE statements. On a synthetic legacy-v1 fixture (L0 install-smoke seeds only plugin_meta + plugin_config + identity), the base tables don't exist yet — migrations crashed with no such table: discussions. Fix: wrap each per-source-table block in tableExists() guards; applySchema's schema.sql re-run is the safety net.
  2. tests/docker/install-smoke.Dockerfile A7 hardcoded schema_version = "2". Fix: read TARGET_SCHEMA_VERSION from db.ts at build time so the assertion auto-tracks future bumps. Backup-file glob broadened to pre-v*.bak.

Verification — ALL layers L0–L4 green; L5/L6 issues filed as follow-ups

Layer Result
L0 install-smoke ✅ PASS (verified after Dockerfile + migration fixes)
L1 lints (26) ✅ PASS
L2 unit (438+) ✅ PASS
L3 integration ✅ PASS
L4 workflow-sim (5) ✅ PASS
L5 dogfood ⚠️ pre-existing fixture drift: stale kind='event' scorers — to be filed
L6 chain ⚠️ 8/10 pass; step 10 = !2927 (consultant prompt re-calibration), step 14 = !2928 (stale workspace sentinel pollution) — both pre-existing, not Phase 2 regressions

pr-reviewer

Verdict: PASS (attempt 2, validation_attempts id=28, loud audit trail with per-claim line citations).

Minor non-blocking note from pr-reviewer: file_registry_embeddings PK omits FK to file_registry (composite-PK source — (repo, path) — single-column FK to rowid isn't portable). Backfill repopulates after any deletion. Backlog candidate: add an AFTER DELETE trigger for orphan cleanup.

Plan after merge (per Human comment on !199 (merged))

After this MR merges, file separate v0.8.0 release-ceremony issue. Both the v0.6.x→v0.7.0-rc upgrade path (v2→v3→v4) and the fresh-DB v4 path are tested in L0 + L2 schema-upgrade tests, so the user-machine upgrade story is verified end-to-end.

Held for manual review per session pattern — no auto-merge.

Merge request reports